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management-or iented information gathering activities were involved: 
"evaluability assessment" (Phase I) and "rapid feedback evaluation" 
(Phase II) tc provide immediate information on program performance* 
The premise fox the study was that it was inappropriate aid unsound 
to attempt a formal program evaluation until an evaluability 
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a rapid feedback evaluation, using readily available information, 
showed positive results* Chapter I summarizes the Phase I 
evaluability assessment and discusses th<e purpose, scope, and general 
approach of the sapid feedback evaluation* Chapters II through VI 
present the results of the five separate assessment activities that 
comprised the rapid feedback evaluation* Chapter VII integrates these 
findings and recommends action steps that can be taken to* improve 
program managemert ^nd operation. (Author) * 



r 



/ 



ERIC 



*****jM** ****** ********** 

* Beproductions supplied by EDRS are the best that can be made *. 

* , from. the original document. * 

* *$*********** ******************************** ♦♦•1f## 4 ******* ************ 



X 



LTV 




AIR-85601-12/81-!FR 



AMERICAN INSTITUTES FOR RESEARCH 
IN THE BEHAVIORAL SCIENCES 

P.O. Box 1113. 1791 Arastradero Rd..Palo Alto Ca 94302 • 415/493-3550 



RAPID FEEDBACK EVALUATION 
OF 

THE W0^EN ' S EDUCATIONAL EQUITY* ACT PROGRAM 

Final Technical Report 



SCOTS OF INTEREST NOTICE 



by 

Peggie L. Campeau 
Dorpthy F. Reynolds 

Joan Druckman 
-Judith A. Appleby 



for 

Organizational Performance Service 
U.S. Department of Education 



The EBJC Facility has* - 
ttusdocumant <ocjroc»e»»| 



JUL 



In otx jotfant»ot.thi«d»a*n^nt 
h mho of interest to the cbmi**- 
houeu nottd to tha » ifhU I ndtw 
infl $houW reflect thtir ipaeal 
points of v»e*r. 



So 



UX DEPARTMENT Of EDUCATION 
NATIONAL INSTITUTE OF EDUCATION 
EDUCATIONAL RESOURCES IN FORMATION 
CENTER (ERJCl 
Tho Ooaxmnt hts bean reproduced as 
received from the. person or organization 
ongmatrng a 

M*kx change* have bean m*de to vnprove 
reproducton quarry 

• Po«ts of view or opinions stated in fv$ docu- 
ment do not necessanfy represent official NIE 
position or policy 



0 



December 1981 



>* This report is made pursuant to Contract No. 300-80-0600. The amount charged 
to the U.S. Department of Education for the work resulting in this report 
(inclusive of -the amounts so charged for any prior reports under this contract) 
Q is approximately £243, ,375. The name 9f the person, employed or retained by the 

contractor, with managerial or professional responsibility for such work, or 
^ for the content of tlve report, is as follows: Peggie L. Campeau, Project Director., 
x 



ERIC 



An Equal Opportunity Employer 



TABLE OF CONTENTS 



LIST OF FIGURES AND TABLES • • • • v 

« 

Chapter I; INTRODUCTION . . . . ; 1 

Overview of the Phase I Evaluability Assessment . 2 

' * J 1 

' Overview of the Phase II Rapid Feedback Evaluation ..... 10 

„ Chapter II: USERS' PERCEPTIONS, OF WEEAP PRODUCTS * ; 15 *' 

Purpose 15 

? Procedure . . .~ • • • • 16 

' . Regjjlts 17 . 

Summary . .* 31 

Implications for WEEAP f s Future Evaluation Planning 32 

Chapter III: ANALYSIS pF WEEAP PRODUCT SALES 35 

Purpose ; 35 1 

Procedure " ' 36 

Results i 39 

Summary . . * 50 

Implications for WEEAP 1 s Future Evaluation Planning J 51 

Chapter IV: IMPACT FROM GRANT PROJECTS 57 

Purpose 57 . 

Procedure 57 

Results 59 

.Summary^ % 72 

Implications for WEEAP \s Future Evaluation Planning 73 

Chapter V: FACTORS INFLUENCING THE UTILIZATION "OF WEEAP 

PRODUCTS AT FIVE NATIONAL DEMONSTRATION SITES ... 77 

Purpose • • 77 

Procedure ' * 78 

Results 81 

Summary » 4 90 

Implications for WEEAP 1 s Future Evaluation Planning 91 

EHJC , -iii- 



t . * ; - : : ■■ — : — : ■ 

* , - 

Chapter VI: SEA PERCEPTIONS OF WEEAP IMPACT 93 . 

Purpose ...*...*. ■ . . . • • 93 

Procedure • • \ \ 94 

Results >...'..... Q7 

* • 0 Summary ^ • -^X3 

Implications for WEEAP's Future Evaluation Planning . . . . 1 113 

r- $ 

Chapter VII: INTEGRATED FINDINGS FROM THE RAPID FEEDBACK 
EVALUATION AND RECOMMENDATIONS FOR IMPROVING 

PROGRAM MANAGEMENT AND EVALUATION ........ 115' „• 

Purpose ■ t 115 

- * Integrated Findings . . . ' ...... 116 

r Recommendations for Improving Program Management , 

"* • * and Evaluation ^3 

\ 

* * *. 
REFERENCES 1 " !3! 

' APPENDIX: " 

» 

j Products Purchased That* Were Discussed by Interviewees - 1 

^ in the User Survey (Chapter II) 135 ' 

Positions of Staff Interviewed at the Five National 

Demonstration sites (Chapter V) 136 



> 

t 

i 

% 1 

4 

i 

ERIC 



LIST OF FIGURES AND TABLES ' 



* Figures 

^ " * »* 

1. Detailed Logic Model of the Intended WEEA Program (follows page 4) 

2. Summary of WEEAP's Field Objectives, Evaluability, 

and Evaluation Questions 8 

3. Overview of Foci for Data Collection and Analyses 

for the WEEAP Rapid Feedback Evaluation ....... 13 , 

4. Correlation of Past and Likely Future Involvement 

in Educational Equity ' . . . Ill 

» 

Tables 



' 1. User Survey Sample by Region and Type of Purchaser . . . : t 16 

2. Distribution of Types of WEEAP Prpducts Discussed 

in the User Survey by Type of Purchaser 18 

* 3. Affiliation and Position of Product Purchasers 

in the User Survey 19 

4. Sources of Information About WEEAP Products by Region ... 20 

5. Additional Sources Consulted for Equity Materials 

by Purchasers in the User Survey Sample . ^ *. . . 21 

6. Groups With Which the* ftEEAF Materials Were Used \ 22 * ; 

7. Outcomes Expected and Accomplished By Use of 

.WEEAP Materials *. . . . ; . 24 

8. Indicators of Product Impact < 25 

9. WEEAP Product Purchasers' Preferences for 

Additional Materials 29 • 

• . 

10. Overall Sales Activity of the WEEAP Publishing Center n 
Based on Yearly' Sales Records •- 40 

11. Most Frequently Purchased Items Based on Yearly 

Sales Records . s I 40 

12. -Most frequently Purchased Items Based on a 5% ^ N 
Random Sample of Invoices (N»388) . 41 • 

13. Types of Items Available Over the Three-Year Period 

1978-1980 . \ . 41 % , 

14. Average' Monthly Sales Per Item by Item Type ^ - » 

Based on Sales Records 42 

15. Comparison of Items Sold and Items Available 

as of the End of 1980 ........ 43 

16. Most Frequent Purchasers 

> Based on a 5X Random Sample of Invoices (N»388) 44 



'17. Regional Variations in Amount of Purchasing 

Based {on a 52 Random Sample of Invoices (N»388) 46 

18. Number* of Items in Three Price Categories 48 

19. Number of Items in Each Price Category 

for 1978, 1979, and 1980 48 

20. Total Item Sales by Price Category % 

for*"the Three-Year Period 1978-1980 . . 49 

21. Brief Descriptions of Sample Projects/Products 

in the Study Sample (N-9) . . . . 58 

22. Types o£ Evaluation Design features 60 

23. Estimated Level pf Effort Devoted to Evaluation 

of Grant Projects . . * 61 

*■— 

24. Number of "^Grantee Evaluations That Address Outcomes 

WEEAP Expects from Grant Projects/Products ........ 62 

25. Expected and Accomplished Outcomes of Grantee Projects . . t. 63 

26. Indicators Used in Grantee Evaluations . . . ' 65 

27. Types of Reported Ripple Effects . * 67 

28. Factors Contributing to* Pro j^fct /Product Impact ....... 6^ 

29. Factors Inhibiting Project/Prpduct Impact 70 

30. v Ifecommendations for Improving Grantee Evaluations . . . . T 71 

\ 

31. Preferences of Demonstration Site's for 

Future Product Development * 87 

32. Sample of Nige States by Selection Category and Region • . 94 ' 

33. Positions. of Interviewees - 96 

,34. Outcomes Intended from the SEAs 1 WEEAP Grant Projects ... 98 

35. Unforeseen Additional Outcomes from SEAs 1 

WEEAP Grant Projects *. 99 

* . . - ■ ; 

36. Post-WEEAP Grant Activity 100 

37. Evidence of Impact from SEAs 1 WEEAP Grants . 10J/ 

38. Outcomes Expected and Accomplished by SEAs 

from Using WEEAP Materials , . 104 

s • 

39. Degree o* Present ^Dependence on WEEAP Materials 

by State Agencies Planning to Continue to .Use 

, . WEEAP Products ' . . 106 

. 

40. " Likely Level of State Educational Equity Activity 

in the Absence of Federal Funding " 106 

41. Indications of State Activity/Involvement in # ' 
Educational Equity and Likelihood of Future Involvement . . *109 



ERLC 



-vtg 



. Chapter I 



INTRODUCTION 



.This report presents the results of a rapid feedback evaluation of the 
extent to which the Women 9 s Educational Equity Act Program (WEEAP) is making 
progress toward its objectives. The study was the second phase of a two- 
. phase assessment carried out by AIR under contract to Organizational Per- 
formance Service (OPS) of the U.S. Department^ of Education. Two different 
kinds of mahagement-oriented information gathering activities were involved: 
"evaluability assessment* 4 (Phase I) and "rapid feedback, evaluation" (Phase 

II) to provide immediate information^on program performance. 

s; 

The premise- for the study was that it was inappropriate and unsound to 
attempt a formal program. evaluation until an evaluability assessment ljad 
Shown WEEAP was capable of functioning as intended and a rapid feedback 
evaluation, using readily available information, showed positive results* 

AIR completed the Phase I evaluability assessment in approximately six 
months. It is described in a report entitled, " Evaluability Assessment of 
the Women' 8 Educational Equity Act Program. Phase I Final Technical Report 
(Campeau, 1980). The present report covers Phase II, the rapid feedback 
evaluation of WEEAP. 

The remainder of Chapter I summarizes the Phase I evaluability assess- 
ment and discusses the purpose, scope, and general approach of the rapid 
feedback evaluation. Chapters .,1*1 through VI present the results of the 

five separate assessment activities that comprised the rapid feedback . * 

f 

evaluation. Chapter VII_£ntegrates these findings and recommends action 
s.teps that can be taken to improve program management and operation. 



A 



Overview ofr the Phase I Evaluablllty Assessment 

The purpose of the evaluablllty assessment was to arrive at a program 
model of WEEAP that was evaluable, i.e., one that: 

• had clearly defined and measurable objectives; 

• had a discernible logic for reaching* the objectives; 

• 'had. a fcet'of sequenced activities that were plausible repte- 

- sentations of the logic; > 

• ♦ * * * 

# either had or could easily obtain feasible process and out- 
come measures; 

# contained clear provisions for program management's and 
• policymakers 1 use of the information provided; and 

# provided options for possible alternatives for improving 
program performance* * 

We used a variety of strategies to arrive at a complete description of 
WEEAP and to complete our preliminary analyses of its evaluablllty* High- 
lights^ of these procedures and findings are presented next. 



A Description of WEEAP «Ag It Is Intended to Operate 

Our initial description of the program was based on studying the WEEAP 
legislatioij*, regulations, an4* other relevant documents and then augmenting 
this with extensive interviews with the WEEAP Director, agency officials to 
* whom she reports, staff at the National 1 Advisory Council on Women's Educa- 
tional Programs (NACWEP), Congressional staff, and constituents who are 
vitally concerned with the program's operation and accomplishments* 

The authorizing legislation * T|ie Women's Educational Equity Act of 
1974 authorized a program of discretionary grants and contracts whose pur-* 
pose was to promote educational equity for women and girls at all levels of 
education* According to the legislative .mandate, this was to be accom- 
plished by funding the development, dissemination, and demonstration of 
model projects, materials, and approaches thatiwould target on every area' 
of education that per^eEuates sex bias* The model products and approaches 
were expected to be useful in a variety of situations and under a variety 
of contritions* The emphasis on variety in the approaches represented in 
these model products reflected the assumption that there are multiple paths 

-a- 8 - 
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to achieving educational equity ^tnd that a range of 'options should ttejpade 
avail&ble. to potential users* A^fu^ther assumption was that widespread use 
of WEEAP products would enhance educational equity and improve education 

for women and gi>rls of all ages. 

( 

The Act was reauthorized and substantially revised by the Education 
Amendments of 1978. A two-tier funding strategy was introduced. Tier 1 
continued the .funding of developmental, dissemination, and demonstration 
activities within the same broad areas' as the original Act. Tier 2 author- 
ized a new program thrust that) would- provide financial assistance to enablto 

• - * * 

educational agencies and institutions and other eligible applicants to meet 

the tequirements of Title IX of the Education Amendments of 1972. Tier 2 
. activity would be triggered automatically once WEEAP f s appropriation 
Exceeded $15 million. An underlying assumption was that the model proj- ^ 
ects, materials, ancf Approaches developed by grantees under Tier 1 would be 
available for use by local school districts and other eligible applicants 
funded under Tier 2. • 

Another major change in the 197& Act was a requirement that the Com- 
missioner (Secretary) establish priorities to ensure 0 the most effective use 
of the funds available. Tfw WEEAP regulations in their final form estab- 
lished five priorities for model projects. The first was .for model pro- 
jects on Title IX compliarfce. There were two separate priorities- fox model 
projects to promote educational equity for racial and ethnic minority women 
and girls, and for disabled women and girls. The fourth priority was for 
projects to influence leaders in educational policy and administration. 
The fifth priority was for model projects to eliminate persistent barriers 
to educational equity for women. In addition, other authorized. activities 
could continue to be funded in a separate category. . — 

Together, the 1978 Act and the regulations establishing priorities for ' 
model projects placed more specific emphasis than ever before on educational 
equity and institutional change. 

„ . »WEEAP f s funding history * WEEAP* s funding* hi story from the 1975 fiscal 
— ; \ 

year through the 1981 fiscal year was as follows: ^ 
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Year { Authorization Appropriation 

1975 $ <^-0-' $ -o« 
* 1976 - 30,000,000 $,270,000 

1977 30,000,000 . 7,270,000 

1978 . 30,000,000 . 8,085,000 

1979 30,1)00,000 9,000,000 

1980 80,000,000 10,000,000 

1981 80,000,000 8,125,000* " 



Altfiougtf'Congress dramatically increased the authorized level of funding 
for the program for the 1980 fiscal year from i3t) million to $80 million (an 
increase of 267 percent), the amount of money actually appropriated fqr WEEAP 
in PY80 represented .only a minimal increase* in funding <from $9 million- to* 
$10 njillion, or an increase of approximately 11 percent)* Thus, WEEAP has 
had to attempt t;o maintain its momentum and to show progress toward its 
objectives it a consistently low appropriation levelr-'too low to trigger the 
implement at ion of Tier £ activity* ^ 

A portrayal of WEEAP in chart form * In an evaluability assessment the 

description, of how the- program is ftitended to operate is rendered in the form 

ft 

of '.a diagram that shows program activities and their intended effects* Galled 
a "logic model, H this portrayal represents the expectations for the program 
stated as. a series of w i£-then H hypotheses* The logic model for WEEAP is 
shown in Figure 1 on the fold-out following this page* The model shows only 
Tier 1 of the program because WEEAP has* never been funded in excess of $15 
million, the trigger level fQr the implementation of Tier 2 activity. 

ft • 4 ' 1 * 

Consensus on Program Objectives 

. , * l § - 

The ultimate goal set for WEEAP by^its bfoad 'legislative mandate is to 

reduce .sex stereotyping and bias in educatidn for females of -all age** The 

logic model in Figure 1 shows the strategies and intermediate outcomes which 

WEEAP management and policymakers regarc} as preconditions f£r Successful 

achievement ^of this ultimate goal* These intermediate outcomes, in turn, 1 

were useful in •clarifying and gaining consensus on a more specific set of 

objectives against which to, measure program performance* 

. ' — '* 

*-When Congress approved the Administration's decision to cut the Depart- 
ment of Education's budget for the 1981 fiscal year -by 25%,,WEEAP , s 
budget (in the Department's continuing resolution) for 1981 was cut by 
12. 5Z from $10 million to $8,125 million. 



INPUTS/RESOURCES 



OUTCOMES 
(federal) 



Program funds are allo- 
cated to WEEAP by 
Congress 



Salary and expense funds 
are allocated to WEEAP 
by the Department of 
, Education, Office of Edu- 
cational Research and 
Improvement (OERD* 



*0ffice of Elementary and 
Secondary Education (OESE), 
since May 7, 1981 



National Advisory Council on Women's Ed- 
ucational Programs (NACWEP) 

a. reports to/advises policymakers 

b. recommends allocation of .funds 

c. recommends criteria for establishing 
'WEEA Program priorities 

.d evaluates WEEA Program 



Women's Educational Equity Act Program- 
staff of ^Department of Education. 

a. provides national leadership 

b. issues regulations and guidelines for 
Program impjementation 

y:. establishes Program priorities and alloca- 
tion o\ funds 



carries out grant selection process on the 
oasis of quality/diversity criteria 
(NACWEP recommends awards be made 
by June) 



g. 



e. monitors- performance of grantees and 
contractors 

f. establishes planning and reporting 
requirements for grants and contracts 
provides grantees with a source of tech- 
nical assistance jn product development 
3rid revisiop (via the Publishing Center^ 
andWEEAPstaffJi ~\ 
r_ ovides grantees with a source of tech- 
nical assistance in grants management and 
with mechanisms for coordinating with 
fellow granjees (via a coordination con- 
tractor and WEEAP staff) 
provides a source for publishing end mar- 
keting products (via the Publishing 
Center) 

provides for showcasing WEEAP products 
and strategies infused in comprehensive 
ways in elementary and secondary educa- 
tion programs (via contracts to establish 
national demonstration sites) 



Successful grant applica- 
tions reflect* 

a. priorities 

b. diverse needs 

t. diverse populations 

d. all educational levels 

e. diverse change strate- 
gies and activities 

f. geographic diversity 

g. different types of 
grantees 



l- 
l 



Other major^on-WEEAP change efforts as 
well%s Zeitgeist V 
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NOTE: This Logic Model describes tKe WEEA Program as Implemented in 1980-1981. 
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Detailed Logic Model of ffife Int 

SOURCES: WEEAP Director, NACWEP Director^ACWEP Evaluation, RFPs, Regulations for 



PROCESS 
(grantees) 



OUTCOMES 
(grantees) 



8 



to 



11 



12 



Grantees develop and test 
model projects, materials, or 
approaches on Title" IX com- 
pliance for use at eveny l?vel 
of education and by various 
.✓groups 



Grantees develop and test 
model projects, materials, or 
approai'js on educational 
equity for racial and ethnic 
minority women and girl$ 



Grantees develop and test 
model projects, materials, or 
approaches orf, educational 
equity for disabled women 
and girls 



7 



Grantees develop and test 
model projects, materials, or 
approaches to influence 
leaders in educational policy 
and administration 



Grantees develop and test 
" model projects, materials, or 
approaches to eliminate per- 
sistent barriers to educational 
equity for women, e.g., physi- 
cal education, vocational 
education, educational admin- 
istration 



Grantees develop and test 
projects, materials, and 
approachesjne other issues of 
national, statewide, or general 
significance to address educa- 
tional equity 



1 



13 



14 



16 



16 



17 



18 



18 



Target groups, staff , and rele- 
vant others benefit from 
exposure to the model 
projects, materials, *or 
approaches *■ 



Conducive environment for 
change and ability to generate 
change are fostered 



Capability of educational sys- 
tems to work for and achieve 
educational equity is 
enhanced 



Educational leaders imple- v 
ment the model projects, 
materials, or approaches 



Diverse model products and 
change strategies are produced 



Evidence of effectiveness 
from grantees' tryout of 
model projects, materials, or 
approaches is available 



Grantees' performance is 
improved through technical 
assistance 



28 |«r — — — — — 

...—I Informal * dissemination 
j activity „ | 

Ummm — -J 



21 


Some products are rejected 
on the basis of screening 
criteria \ 


20 






Products and supporting 
evidence of effectiveness 
are received and reviewed 
by WEEAP's Publishing 
Center contractor 


► 






f 


22 


Some products are revised 
by grantees or b</ WEEAP's . 
Publishing Center con- * 
tractor 



Figure 1 



I of the Intended WEEA Program 

RFPs, Regulations for Implementing the WEEA Legislation, WEEAP Publishing Center Contractor 



PROCESS 
(dissemination) 



OUTCOMES 
(potential users) 



«r 1 

^| Informal dissemination I, 
1 activity. | 

J 



26 



25 



Some products are rejected 
on the basis of screening 
criteria 



20 



Products, and supporting 
evidence of effectiveness 
are received and revjewed 
by WEEAP's Publishing 
Center contractor 



22 



23 



Some products are revised 
by granteesor by WEEAP's 
Publishing Center con- 
tractor 



27 



JDRP-approved WEEAP 
products become eligible 
for National Diffusion 
Network (NON) dissemi- 
nation funds (competitive 
award process) 



Some WEEAP products 
are submitted to the Joint 
Dissemination Review 
Panel (JDRP) 



Those products judged 
exportable for national 
distribution to potential 
users are produced and 
marketed 



24 

— 



WEEAP products are pur 
chased by target groups 



29 



WEEAP-funded demon- 
stration sites "showcase' 
WEEAP products 



Educational agencies/insti- 
tutions and other groups 
respond favorably to 
WEEAP products 



PROCESS 
(ut«rt) 



LONG TERM 
IMPACT 




s/insti- 
oups 



30 



32 



33 



34 



35 



36 



Educational agencies/insti- 
tutions and other groups 
use^EEAP products 
nationwide in educational 
programs, preschool to 
adult 



Users find WEEAP model 
products and approaches to 
be usable, useful, valuable, 
beneficial, adaptable 



Users find technical assistance 
in implementation of WEEAP 
products and approaches to 
be adequate 



Conducive environment for 
change and ability to gerr*ate 
change are fostered 



Capability of educational 
systems to work tor and 
achieve educational equity is 
enhanced 



37 

— 



6. 



Sex ^te'reoty ping and bias 
are reduced in educatibn 
for females of all ages * 



Educational leaders imple- 
ment the model projects, 
materials, or apprcdches 



Target groups, staff, and 
relevant others benefit from 
exposure to the model 
projects, materials, or 
approaches 
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The agreed upon specific objectives for WEEAP's federal level and 
field operations are listed below* 
Federal level objectives ; 

• Ensure that 'the gran** selection process is fair and con- 
sistent. 

• Ensure that Prograd Officers facilitate successful comple- 
tion of grant projects, particularly with respect to identi- 

* fying problems early and participating in their resolution. < 

Field level objectives : 

• Develop diverse, tested model product^ and change strategies. 

• Produce and market the best of these model products and 
strategies to potential users, nationwide. 

• Demonstrate that these model product3 and strategies are 
i usable, useful, valuable, beneficial, and adaptable* 

r 

• Specifically, demonstrate that use of the model products and 
strategies: 

- produces positive changes in participants 9 behavior, 
attitudes, aspirations, and awareness and in educational 

^ pdlicy and practice; 

- fosters a conducive environment for equitable change; 

- enhances the capability of the educational system to work 
for and to achieve educational equity* 

Our document reviews and federal level interviews in Phase I confirmed 
chere was general agreement that WEEAfc should bQ directing its efforts toward 
accomplishments that are consistent with these intermediate objectives. 

WEEAP's Readiness for Evaluation 

A major purpose of the Phase I evaluability assessment was to determine 
whether WEEAP's objectives were plausible and measurable at present and 
whether further evaluation of them in the Phase II rapid feedback evaluation, 
was feasible and likely to be useful* Ojar methods of making these determi- 
nations included further extensive interviews with the WEEAP Director and 
her staff f federal level policymakers, staff at the National Advisory Coun- 
cil for Women's Educational Programs (foACWEP), Congressional staff, key staff 
at the WEEAP Publishing Center, and staff of grant projects at nine sites* 



\ 



Our alms In all of these contacts were the same: (1) to clarify what was 
really being attempted with respect 4 to a particular objective or activity; 
(2) to Identify points at which program performance could be assessed with 
respect to WEEAP' s agreed upon objectives; and (3) to determine what mea- 

sures or indicators of accomplishment were readily available in the event 

< » 

that the Phase II rapid feedback evaluation was deemed desirable. 

Conceivably, an evaluability assessment could reveal that there Is no 
real program-- that the rhetoric in Washington was not backed up by opera- 
tions in the field that could possibly produce the outcomes being promised 
by the legislation, program management, and spokespersons concerned with fhe 
program's continuance. In WEEAP* s case, the evaluability assessment showed 
a more positive result. We found that there was reasonably close correspon- 
dence between the model of. the intended/ program and the program in place, 
that is, its day-to-day operation. 

Plausibility of WEEAP 's objectives . In assessing WEEAP's readiness for 
evaluation, we made judgments about the necessary and sufficient conditions 
that must be in place for WEEAP to achieve its objectives. In the language 
of evaluability assessment, making such judgments Is called determining the 
plausibility of the program's objectives.* These judgments involved estimat- 
ing the extent to which present conditions justified expectations about 
WEEAP's ability to achieve its objectives. The results of our analysis are 
presented in the report on Phase I of this study (Campeau, 1980). In brief, 
we concluded that, while there was insufficient justification at WEEAP's 
present level of funding for expecting desired changes in educational equity 
to have been accomplished on -an impressive scple, this is not to say that 
WEEAP is unable to show progress toward its objectives. 

Feasibility of measuring program performance . Assessing WEEAP's readi- 
ness for evaluation also entailed judging whether the agreed upon objectives 
were measurable and whether feasible sources of performance data were avail- 
able to indicate progress in achieving them. We concluded that WEEAP's 
objectives were measurable , but that: 

• data on user response to WEEAP products were- not available 
because no systematic 'user survey had been done; 

. * • 

• evidence of impact from grant projects and products was 
likely to be subjective and informal and might not neces- 
sarily address outcomes of most interest to WEEAP; and 
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• quantitative information that would support what previously 
* have been largely impressionistic statements about WEEAP 
product dissemination was lacking* 

We^ further concluded feasible data sources existed thpt could be 

accessed tp obtain this information (e.g., the WEEAP Publishing Center's 
sales records and state-by-state files of purchase invoices identifying the * 
names and locations of users, grantees 1 evaluation reports). 

Utility of program performance data . The final purpose of the Phase I 
evaluability assessment was to determine whether the information obtained on 
WEEAP's progress toward each of the agreed upon objectives would be used by 
program management an<J policymakers. Figure 2 displays each of the agreed 
upon objectives for WEEAP's field operation* and, alongside each, our judg- 
ments of WEEAP f s readiness at present to show progress toward the objectives, 
and" the pertinent evaluation questions. These evaluation questions^ reflect 
the consensus of program management, policymakers, and the Work Gro\ip** on 
the type of information with highest utility at present for program planning 
and for demonstrating *WEEAP r s success in achieving its objectives* * 



* The agreed upon objectives for ^EAP's federal level operation, listed 
earlier, do not appear in Figure 2 because it was determined to exclude 
their assessment in Phase II 1 s rapid feedback evaluation. This decision 
reflected program management's uncertainty about the extent of future 
authorization and appropriations required to continue WEEAP as a federal 
program. The consensus was that the mo$t important, appropriate, and 
economical focus for Phase II would.be WEEAP's accomplishments 'in devel- 
oping, disseminating, and demonstrating the usefulness and impac£-qf its 
products, projects, and approaches* 





** Phase I was carried ou£ by AIR in close cooperation with a Work Group 
that met every two or three weeks during the first .five- months of- the 
contract* The Work Group^ consisted of the WEEAP Director, the AIR Proj- 
ect Director, the OPS Project Officers, and„other Department of Education 
staff. The AIR Project Director kept the Work Group informed of the 
study's procedures and findings. The Work Group advised and assisted in 
identifying and accessing data sources, identifying measures and indica- 
tors that could be used in' a subsequent assessment, and developing man- 
agement and evaluation options. 
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Figure 2 

SUMMARY OF WEEAP' S FIELD OBJECTIVES, EVALUABIL1TY, AND EVALUATION QUESTIONS 



• * 

; — ■ — 1 


• 






5 . % -f * .. 

Program Objective 


<* 

WEEAP's Readiness for Evaluation 
(Campeau, 1986) 


1 " 

Evaluation Questions 


• Cross-Ref • 
.Code for 
Phase II 
Assessment 


# Develop diverse, tested S 
' model products and 
f change strategies • 


This objective can be met as* WEEAP is 
currently operating. However, the 
definition of "tested 11 varies from 
"expert review" to "field testing" of 
model products and change strategies. 


What evidence exists of the 
tested effectiveness of 
WEEAP products and 
strategies? \ 


A 


• Produce and market the 
best of these model pro- 
ducts and strategies to 
potential users, nation- 
wide . +■ 

"do. 
t 


This objective can conceivably be jnet as 
WEEAP'a Publishing Center is operating,*. 
However, the budget is too limited to 
support aggressive marketing or in-depth 
assistance to users of WEEAP products 
and strategies. 


What .evidence exists to 
portray convincingly 

the efficiency and effec-v 

tiveness of WEEAP's pro- 
duct dissemination, effort? 


B 


• Demonstrate that these 
m /model products and 

strategies are usable, 
useful, valuable, 
beneficial, and adaptable* 

- 


The plausibility of this objective cannot 
be estimated at present because an 
adequate tiser survey has not been done 
and because thg national demonstration 
sites are still in their- initial .phases 
of implementing WEEAP products and * 
strategies outside the developer sites. 


Are WEEAP products found to 
be of high quality, easy to 
implement, adaptable, use- 
ful, and effective by those 
* Who purchase them? What 
factors influence imple- 

.ii-i — £ /-» m J\ ** 4*^ linn t*i nn 

men ta tipn ana u uxuza cion 
of WEEAP products and 
strategies? m • 


C 




% 


* 
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Figure 2 (continued) 
SUMMARY OF WBEAP'S FIELD OBJECTIVES, EVALUABILITY, AND EVALUATION QUESTIONS 



I 

I 



Program Objective 



• Specifically, demonstrate 
that use of the model 
products and strategies; 

- -produces positive 
changes in partici- 
pants 1 behavior, 
attitudes, aspira- 
tions, and awareness 
and in educational 
policy and practice; 

- fosters a conducive 
environment for 

' equitable change; 

- enhances the capabil- 
ity of the educational 
system to work for 
and to achieve educa- 
tional equity. 

s 
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WEEAP's Readiness for Evaluation 
r (Campeau, 1980) • 




The plausibility, of this objective cannot 
be estimated in the absence of objective 
assessment. Although grantees in our 
study sample for the Phase I evaluability 
assessment said that these effects were 
gratifying aspects- of their projects, 
only informal evidence existed in most 
cases to attest to their actual occur- 
rence.. Similarly, no objective evidence 
of effects has been sought from users 
of WEEAF's model products and change « 
strategies j 



Cross -^Ref ♦ 
Code for 8 
Phase II 
Assessment 



What is the nature and con- 
text of product impact? 
What indications are there 
that use of WEEAP products 
and strategies can contrib- 
ute to promoting education- 
al equity and . improving 
education? 
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Overview of the Phase II Rapid Feedback Evaluation 

1 The remainder "of this chapter explains the purpose, scope, and general 

* » 

approach used in the rapid feedback evaluation. 

Purposed * . * 

The purpose of the rapid feedback evalution was to: 

• obtain *a rough estimate of progres* toward the agreed upon / 
objectives for the'WEEA Program from information that was 

easy to obtain and analyze; 

• estimate the difficulty and likely utility of conducting a 
more formal .evaluation of program performance in the future; 
and 

• recommend action steps (management and evaluation options) 
that could be exercised to improve the program. 

The intended audience for this information was program management and 
policymakers*" The intended use of the information was to maintain or adjust 
program^ activities 
some future time. 



program activities, and to plan a formal evaluation If that were desirable at 



Scope , %% 

The rapid feedback evaluation's scope was determined by the foregoing 
results of the evaluability assessment, as summarized in Figure 2 on pp. 8, 9; 
Specifically, the Phase II rapid feedback evaluation consisted of five sep- 
arate assessments. They are listed below and cross-referenced with a letter 
code to Figure 2. This format is intended to qlarify the connection between 
Phase II and the program objectives and evaluation questions agreed on in 
Phase !• The H chapter reference* serves as an index to the results of the 
rapid feedback evaluation, presented in the next five chapters. 
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Nature of the/ Phase II Assessment Activity 

Determine whether users perceive WEEAP 
products toyoe of high quality, easy to 
implement, adaptable, useful, and effective* 

* / 

Carry out /a limited number of sales 
analyses Jbased on readily available \ 
information from the WEEAP Publishing 
Center. / 

Obtain and synthesize readily available 
evidence of impact from grantees 1 product 
and project implementations* • 

.Identify factors that have influenced the 
utilization of WEEAP products and approaches 
in comprehensive educational programs and 
describe their consequences* 

• > 
Determine whether state education agencies 
(SEAs) that have received WEEAP grants or 
used WEEAP products found that these* 
activities contributed to improved edu- 
cational programs, policies, or practices. 



Figure 2 
Reference 
( see pp. 8, 9 ) 



B 

A, D 



Chapter 
Reference 



Chapter II 



Chapter III 



Chapter IV 



Chapter V 



Chapter VI 



iJ 



General Approach 

The methodology for rapid feedback evaluation emphasizes the u«* * ' 
* small samples* and the collection of readily* available qualit. ; f/v* Quan- 
titative data that is directly pertinent to assessing program performance on 
agreed upon objectives* 



* There are caveats with respect to generalizing findings from each data 
source, and these are carefully explained in each of the next five chap- 
ters. In brief, it is not within the scope of a rapid feedback evaluation 
to provide definitive answers to questions about WEEAP 1 s product develop- 
ment, dissemination, and demonstration activities* Instead, the pulrpose 
'is to do what amounts to a "pilot evaluation." This provides very rough 
estimates of urogram performance and permits -us to make educated recommen- 
dations as to what should be examined in future, formal evaluations, what 
data sources and indicators of accomplishment can most profitably be' used* 
how such data might be organized and reported, and so on. A rapid feed- 
back evaluation also helps to determine where further evaluation is not 
feasible,' or whfere the quality and substance of data will not be worth the 



cost. 
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Information for WEEAP' s, rapid fee^oack evaluation was obtained from 

five sources: > ^ . 

\* • 25 product users, about equally divided among postsecondary 

V „ institutions, state education agencies (SEAs), local educa- \ 

tion agencies (LEAs), and o.thers (see Chapter II); 

• sales records for 1978, 1979, and 1980, and 388 purchase 
invoices representing a 5% random sample of those on f il* at 
the WEEAP Publishing Center (see Chapter III); 

J 

nine grantees whose products were thought by the^WEEAP Pub- 
lishing Center to be among the highest quality and best 
selling materials to have been produced and disseminated as 
* of the end of 1980 (see Chapter IV);- 

• staff for the five national demonstration sites where WEEAP 
products and approaches are being used in a comprehensive 
way to implement sex-fair educational programs (see Chap- 

. ter V); and 

» 

• nine state education agencies that had either been WEEAP 
^grantees, recipients of Civil Rights Act Title IV funds* for 
sex equity activity, or frequent purchasers of WEEAP prod- 
ucts (see Chapter VI), / p 

Data collection methods related to each of the above sources are 

» ■* 

described in the appropriate chapters. Briefly, our methods consisted of 
irrdepth interviews, site visits, and computer analyses of sales records and^ — 
coded information on purchase invoices* 

There were many points of overlap in* the data elements that composed 
the foci of these separate data collection activities. This was inten- 
tional and was a strategy for hedging against the limited general! zability 
of findings from any single data source^ 

Figure 3 portrays the nature and extent of this overlap, and provides a 
graphic representation of our strategy for integrating findings across data 
sources that provided information on< similar points* Chapter VII presents 
the integrated finding* on WKEAP's progress toward achieving the agraed*upon 
objectives* It also sets forth management and evaluation options that can 
be implemented to strengthen the WEEA Program and to improve its ability to 
demonstrate measurable success in meeting its objectives* 

24 
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Figure 3 

V 

OVERVIEW OF FOCI FOR DATA COLLECTION AMD ANALYSES FOR THE VEEAP RAPID FEEDBACK EVALUATION 



to 



Afi 



3* 



As ? • 



« 2 • 



5 to 



2f * a 



*3 ^ 



Focus: Product Choice 

• factors Influencing product choice 

• course of *Hllcn if product had not been available 

« » « 

• char ac tar istics of product used 

Focus: Product Utilisation 

• nature of product utilisation 

a tut 'of implementation/ adaptation ^ 

• assistance wanted or ne«<ied In using ptoduct 
. • factors affecting product utilization 

Eocua: Product and/or Project Impact 

• outcomes usar/ gran tea a xp acted 
a outcomes accomplished « 

a indicators accepted as evidence' of each outcome accomplished 
e nature of activities or bonus outcomes after grant project or use of 
produces 

e factors contributing to or inhibiting product or project impact 

Foc us: Scope o* Project and Product Evaluation 
^e purpose of grantee's evaluation 
e congruence or disparity between grantee's evaluation xocua and 

VEEAP' s needs for targeted evaluation dace 
e ltvel of effort devoted to project and product evaluation vls-a-vis 
amount* duration, and purpose of grant 

Focus: Recommendations for Improvini Project and Product Evaluation 
e improving quality of evaluations 

e improving pertinence of evaluations for WEEAP 1 s information needs 
Focus: Product Quality 

e purchaser's rating of product's technical quality (printing, 

graphics, format* etc.) 
e purchaser's rating of product's substantive quality (accuracy, 

pertinence, adequacy, etc.) 

Focus: Future Interest/Preferences for Model Pr oducts. Materials, 

Approaches 
e decision re continued use of products 

• preferred directions for product development 

e preferred* delivery system for product dissemination 

9 preferred strategies for "technical assistance in product utiliza- 
tion (includes demonstrations of product u till ration 

• e intereet in continuing or endorsing educational equity activity, 

• in the future v 

• factors encouraging or "hindering educational equity activity 
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X 
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X 
X 
I X 

X 

X 

, X 



X 

X 



X 
X 

X 

X 

,x 

X 
X 
X 

X 
X 



X 
X 
X 



X 
X 



X 

X 
X 

X 



X 
X 



X 

X <* 



Chapter II 
USERS/ PERCEPTIONS OF WEEAP PRODUCTS 

** Purpose 

The- user survey was designed to determine users' perceptions of WEEAP 
products in terns of their quality, ease of implementation, adaptability, 
V usefulness, and effectiveness. AIR conducted telephone interviews with a 

sample of product users to obtain this information, focusing the interviews 
on the following questions: 

i 

Product Choice 

" — " b 
• What type of product was purchased? 
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• Who was responsible for selecting the materials? 

• *What sources were consulted to learn about educational equity' 

materials? 

Product Utilization 

• Were the materials used? 

a 

• What was the target audience with whom the materials were used? 

• How were the materials used? • - 4 - 
Product "Impact t m » 

• Wha£ was the motivation for purchase or the intanded use of 
the materials? 

What impact did the materials have? 

• What indicators were accepted as evidence of this impact? 
#J*Were there any "bonus" outcomes from use of the materials? 

Product Quality 

/- 

• What was the technical and substantive quality of the 
materials? 

Future Directions 

• .What plans, are there for continued use of the WEEAP product^? * 

• What kinds of educational equity materials should WEEAP ' * » 
develop in the future? ' 

% %k 



t 



• , Is WEEAP's current publication and dissemination system 
satisfactory? 



Procedure 

Sampling - - 

A random sample of 220 invoices was selected from the WEEAP Publishing 
Center's files, which contain a total of 7,760 purchase invoices. These 220 
invoices were sorted into four groups by geographic region, and each of 
these was sorted into four purchaser categories: local education 'agencies 
(LEAs) , state education agencies (SEAs), institution^ of higher education 
(IHEs), and Other** A random sample of 32 purchasers was then selected, 
with two purchasers of each type for each region. ' 

When AIR learned that the sample of LEAs and SEAs had to be restricted 
to the nine states cleared for the rapid feedback evaluation, the sample was 
reduced to 28 purchasers and adjusted as is shown in Table 1« The sample 
that resulted from this adjustment was not a statistically random sample* 

*v 1 

t 

Table 1 

< 

USER SURVEY SAMPLE BY REGION AND TYPE OF PURCHASER 
(Showing Number of Interviews Completed 
over Number of Invoices Selected — 25/28) 





* 


Type of Purchaser 






Region 


LEA 


SEA 


I HE 


Other 


Total 


Northeast 


2/2 " 


' 0/0 


2/2 


3/3 


7/7 


Southeast 


• 1/1 


1/1 


2/3 


0/1 


4/6 


Nortl* Central . 


2/2 


1/1 


2/2 * 


2/2 


7/7 


West 


2/2 


2/2 


2/2 


1/2 


7/8 


Total 


7/7 


4/4 


8/9 


6/8 


25/28 



NOTE: Regions were defined as follows: 

Northeast: CT, DE, ME, MB,. MA, NH, NJ, NY, PA, RI, VT , 
Southeast: AL, AR, PL, GA, XT, LA, MS, NC, SC, TN, VA, WV 
North Central: 1L, IN, IA, KS, MI, MN, M0, NE, ND, OH, SD, WI 
West: AK, AZ, CA, CO, HI, ID. MT, NV, NM, OK, OR, TX, UT, WA, WY 




* The- Other category included private companies, county or city govern- 
cants, public and private service agencies, and individuals. 
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Survey Methodology and^Oeneralizability of Findings 

* Telephone interviews lasting 30 minutes to one hour were conducted 
with 25 WEEAP product users — ircluding representatives from seven LEAs, 
four SEAs, eight IHEs, and six Pthers. The small number of interviews 
conducted with SEA purchasers was supplemented by expanding the' interview 
with three purchasers interviewed in another part of this study (reported 
in Chapter VI) • Three other interviews could not be completed because, in - 
two cases, tfce person knowledgeable about product use was not available, 

0 

and in one case, the 'purchaser could not recall making the purchase. 

Estimating the size of the universe of WEEAP product users to be at 
least the numbet of purchase invoices "on file at the WEEAP -Publishing Center 
(7,760 invoices)., findings based on a sample of 25 users yield maximum 95Z 
confidence bounds of approximately 19.6 percentage points. Thus, if in our 
y sample of 25 users^45Z report that they did not use the raterials they 
bought in their entirety but- only very selectively, the true percentage will 
be somewhere between 25% and 65Z. 



Results 

Results of Jche user survey are summarized here according to the five 
focal points of the interviews: 

• Product C&oice 

• Product Utilization ' 

• Product Impact / 

• Product Quality 

• Future Directions > 

♦ 

Product Choice 

What type of product was purchased? Interviewers focused the discus- 
sion on a single WEEAP product, whenever possible selecting one of the items 
listed on the invoice that was made available by the WEEAP Publishing Center 
after January 1979.' Occasionally, interviewees preferred to discuss another 
WEEAP product which they had us,ed and vith which they were more familiar. 
Some wished to discuss acre than one set of WEEAP materials, and one 
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preferred to discuss WEEAP products in general. Table 2 shows the distribu- 
tion of types of products discussed by type of purchaser. 

v 

s 



Table 2 

i 

DISTRIBUTION OF TYPES OF WEEAP PRODUCTS DISCUSSED IN 
THE USER SURVEY BY TYPE OF PURCHASER 

Type of Purchaser 



Type of Product* 


LEA (N«7) 


SEA (N-4) 


I HE (N-8) 


Other (N-6) 


Total 


Staff Development 


8 


1 


1 




10 


Curriculum** 


3 


2 


4 - 


2 


11 


Career Development 


4 


1 


3 


4 


12 


Counseling & Guidance 


1 




1 


1 


3 


Ed. Administration 


1 




1 




' 2 


Unspecified 




i 






1 



* Classified according to listing in the WEEAP Publishing Center's catalog* 
Many materials are classified as being of more than one type, and that is 
reflected by double counting in the table* .See Chapter III for an explana- 
tion of the catalog categories* 

** Includes Early Childhood Education r 

Specific products discussed are listed by title in the Appendix* As can be 
seen in Table 2, the most frequently discuseed products were about equally 
divided between Staff Development, Curriculum, and Career Development 
materials* Most Staff Development materials were purchased at the LEA 
level, but the Curriculum and Career Development materials were purchased 
4 by all groups about equally* 

' Who was responsible for selecting the material? Purchasers repre- 
sented a variety of professional positions, a 3 is shown in Table 3* 
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Table 3 



4 



AFFILIATION AND POSITION OF BJIODUCT PURCHASERS 



IN THE • USER SURVEY „ 



LEAs (N»7) ' f> 

Assistant Superintendent 1 

Assistant Principal, secondary school 1 

Special Projects Coordinator, school district 1 

Project Director, secondary school 1 

Physical Education Consultant, school district 1 

Project Director, nontraditional adult students 1 

Resource Counselor, school district 1 

SEAs (N»4) 

Specialist in sex equity ' * 1 
Specialist in vocational equal educational opportunity 1 

Director, Title IX Assistance Office 1 

Director, Human Relations 1 

IHEs (N-8)* 

Professor or instructor 3 

Director, equity ^project 2 

Director, career services 2 

Director, adult basic education program 1 

* Career - c ounselor 1 

Librarian 1 

Coordinator of women's programs . 1 
'Member, YWCA Board of Directors . . 1 

Other (N-6) 

Owner /director, private company 2 

Director, nonprofit social service agency 1 

Counselor, nonprofit social service agency 1 

University student 1 

Career counselor, self-employed 1 



* The IEE category reflects double counting because some purchasers could 
be classified in more than one affiliation/position category. e 

Of the 25 purchasers interviewed, 52 percent were first-time purchasers 
of WEEAP products. (Twenty-four percent were repeat purchasers, and 24 per- 
cent could not remember if this was a first purchase or not.) 

What sources 'were consulted to learn about educational equity materials ? 
Table 4 shows the. information sources that were consulted by purchasers in 
the user survey sample. The sources of information mentioned most frequently 
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by this sample of purchasers (11/25) were the WEEAP Publishing Center cata- 
log and brochures* Most of those interviewed (16/25) were aware that the 
materials they purchased were originally developed with WEEAP funds. The 
other interviewees (9/25) were unaware of this connection. 



Table 4 

SOURCES * OF INFORMATION ABOUT WEEAP PRODUCTS BY REGION* 
(N«25; double counting possible) 



Source of Information 



Region 

Northeast Southeast North Central West TOTALS 
(N«7) (N»4) (N«7) (N«7) 



WEEAP Publishing Center catalog 
WEEAP Publishing Center * 

brochure/ f 1 ier . 
Booth at conference 
Direct contact with developer 
Other mailed information 
Advertisement in publication V 
SEA workshop X 
Class reading list 
Article In newsletter 
Purchaser was WEEAP grantee 0 
Word-of-nfouth 
School librarian 
Mot known 



1 
1 

1 
1 



2 
1 



3 
1 
3 



3 
3 
. 3 



* there appeared to be no systematic relationship in this sample of purchasers 
between the purchasers' geographic location and their source of information 
about WEEAP products. 

» 

There were two main circumstances under which purchasers bought materi- 
als fronv the WEEAP Publishing Center. The largest number of purchasers 
(44%) were actively searching for materials to meet the needs of a specific 
program, course, or project when they ordered materials from the Publishing 
Center; The next largest group (40Z) received the catalog or other informa- 
tion from the Publishing Center and then purchased materials they thought 
would be of general interest* In addition, a few interviewees made their 
purchases because: (1) the agency where they worked regularly purchased all 
WEEAP products; (2) the purchaser was participating in a WEEAP grant project 
which required use of WEEAP materials; or (3) the purchaser saw the Publish- 
ing Center catalog at a conference and was inspired by it to start a program 
based on the- WEEAP materials. 
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When asked what they would have done if the materials they purchased 
from the WEEAP Publishing Center had oojt been available, 18 of the 25 inter- 
viewed said they would have^used materials from other sources they had con- 
sulted. Table 5 shows additional agencies that were consulted by* this 
sample of 25 purchasers. Many .of those interviewed added that it would he 
unfortunate if W5EAP products were not available, because WEEAP and its 
Publishing Center were their best .source of materials that combine resource 
information with training programs. Two purchasers said they would have 
tried to develop their own materials, and one said that the WEEAP products 
were the primary motivation for their equity project, and, without them the 
whole idea would not have evolved. 

/ Table 5 

ADDITIONAL SOURCES CONSULTED FOR EQUITY MATERIALS BY PURCHASERS 

*IN THE USER SURVEY SAMPLE 
(N-25; double counting possible) 

** - \ 

Source * Frequency of Mention 



Sex Desegregation Assistance Centers 4 

Career education materials developers 3 

Cbuncil on Interracial Books for Children 2 

SEA Title IX Assistance Office >, 2 

U.S. Department of Labor 2 

University of Michigan Continuing Education Center 2 

State Education Agency 2 
Project CARE, Educational. Testing Service , 1 

Arizona State University 1 

The Feminist Press 1 
New York Racism and Sexism Resource Center for Education 1 

Wider Opportunities for Women (WOW). 1 

College Placement Council 1 

APGA Journal 1 



Produ ct Utilization 

Were the materials used? WEEAP materials were used by 72 percent of 



those interviewed (18/25).* The regions where the highest percentage of our 
sample used the products purchased were the Southeast (10QZ or 4/4), and the 
North Central and West (both 86Z or 6/7). Only 28% of the Northeast sample 
used the materials (2/7). 

Almost half of the' purchasers (8/18) who used and liked WEEAP materials 
indicated that they found the materials to be "uniquely suited to (their) 



audience and to the needs at hand.* Others (3/18) were enthusiastic about 
WEEAP materials because of H their attractive, easy-to-access format" and 
. H their clear, nontechnical language." Another factor which prompted pur- 
chasers to use WEEAP materials was that "issues received a balanced treat- 
ment which appealed ; *to both sexes** (2/18). One purchaser commented that 
materials- developed by WEEAP had an automatic credibility which commercially 

C' 

% developed materials did not nebessarily have* Finally, the availability of 

* • * 

rental copies of audiovisual materials was cited as a positive factor in 
product utilization. 

In the cases where the material^ were not used, the reason most often 
* cited' (4/7) for lack of use was an external event > such as curtailment of 
funds or change of job assignment. Other reasons for non-use were that the 
materials were ordered and received too late, that the materials were not 
appropriate to the need, or that the community was conservative regarding 
equity issues* One purchaser could not remember much about the materials. 
Three of the seven non-users plan to use the products in the future. 

Who used the materials? WEEAP products were most often used with 
teachers for inservice workshops or by them as backup material for classroom 
use and course planning* Table 6 shows the groups with which the WEEAP 
materials were used. (Table 6 and the narrative in this and the next sec- 
tion are based on the 18 purchasers who used the WEEAP products they bought.) 

Table 6 * 'J 

GROUPS WITH WHICH THE WEEAP. MATERIALS WERE" USED 
(N-18; double counting because of multiple users) 

Group Frequency 

Teachers 11 

(elementary /secondary) - (10) 

(higher education) (1) 

Students 8 

(elementary/secondary) (4) 

(higher education) (4) 

Adult woaeii ("re-entry" women). 3 

School or project administrators/staff 3. 

Parents and community groups 1 

Librarian t . 1 

Researcher 1 

Job Counselor 1 
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How were the materials used? WEEAP materials were used in one or more 
of three primary ways. Host "frequently, the materials were used along with 
other resources as references to develop educational programs, curricula,. or 
workshops (10/18). Secondly, the materials were used directly in classrooms 
or workshops (7/18). (The' distinction between -these two uses is that in the 
first case the users referred to the WEEAP products to develop their own 
materials or presentations, and in the second, they used part or all of the 
materials as purchased.) The third major use of the materials was to place 
them in a -library or resource center and thus make them available to teach- 
ers, students, consultants, or any. other group (8/18). 

Most purchasers chose to adapt the materials to individual needs, rather 
than use them exactly as purchased (10/18). They also tended to use the 
products in combination with other materials (8/18). However, seven users 
reported implementing the materials more or less as purchased. 

All users found, the products "very easy 1 ' to use," and only one mentioned 
that technical assistance in product use would have been welcome. The pur- 
chasers interviewed gave two main reasons for not needing technical assis- 
tance. First, most of them considered themselves to be sufficiently experi- 
enced in teaching, conducting workshops, etc., so that they cotria" use the 
materials without outside "help. Second, they said the materials themselves 
were well organized and self-explanatory, and thus could be easily imple- 
mented even by inexperienced users. 

Product Impact 

What was the motivation for purchase, or the outcome expected from use 
of the materials? In general, most purchasers expected the WEEAP materials 
to raise the awareness about educational equity issues in the group with 
which they were used. We asked each of "the purchasers what they expected to 
accomplish by using the WEEAP materials and whether, in ^heir judgment, they 
had achieved their expectations. Table 7 shows the various specific out- 
comes expected and accomplished. 



. Table 7 

OUTCOMES EXPECTED AND ACCOMPLISHED BY USE OP WEEAP MATERIALS 

(N»25 interviews) 

i 

Frequency 
Ac com- Not 

Outcomes . ^ Expected plished Not Sure Used 

Knowledge ; af and concern for equity among 

echool administrators and faculty enhanced 9 ' v 7 2 

A "women 1 8 program 1 ' in the school or 

community implemented 4 1 3 

Sex stereotyping in students' career 

options reduced or eliminated 3 2 1 

Students 1 awareness of sex stereotyping 
heightened and a commitment to act to 

stop it instilled * 2 2 

Women assisted to re-enter school 'or 

job market 1 i 

Older women assisted to enter job market 

for first time , 1 1 

Curriculum units (secondary level) on 

equity developed 1 1 

Handbook ta nontraditional courses developed 1 l 

Research materials on equity for minority 

women developed i i 

Resume-writing skills of purchaser, a 

self-employed career counselor, improved 1 l 

Bilingual education library augmented 1 1 



What impact did the materials have? In all cases where the materials 
were used, purchasers either thought the materials had helped them maet thei. 
goals and that the* desired impact had resulted (13/18) or they said they had 
insufficient evidence to determine the impact of the materials (5/18). No 
one said that the materials were used but did not bring the desired result. 




What Indicators were accepted as evidence of Impact? Table 8 shows the 
kinds of indicators accepted by the 18 users as evidence of the usefulness 
of the WEEAP products. The impression that most users had that the WEEAP 
materials helped them meet their goals' was by and large based on 'informal 
comments from those involved in product use. Some sort of quantifiable 
evaluation data was gathered by. less than one-third of the 18 product users 
(5/18), and aven these data could not be construed as representing an 
assessment of the impact of the WEEAP materials per se. This is because 

(1) the data collected were meant to assess only one aspect (e.g. teacher 
attitude) of a larger equity program in which the materials were used; or 

(2) they measured overall impact of the program (workshop, class, etc.) in 
which the materials were used in combination with other ncm-WEEAP-developed 
materials and approaches. Thus, even where data were gathered, it was 
impossible to isolate the impact of the WEEAP products. 

Table 8, 

INDICATORS pF PRODUCT IMPACT 
(K-18; duplicate count because of more than one indicatorvper site) 



Indicators 

General impressions of user-purchaser 

Participant evaluation of workshop 

Classroom or audience- participation observed by user 

Uncompiled data on numbers of requests for materials 

Follow-up reports by staff on those involved in product use 

Pre-post attitude survey 

Uncompiled data on class enrollments 

Uncompiled data on numbers of contacts made by equity program 

Uncompiled data on number of equity programs set up after 
consultation wirh product user 

Rating of SEA equity programs by Tiile IV-C Office, 
Washington DC 

Invitations to equity project staff to make other 
presentations 



Frequency , 
5 
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The following examples of product use and evaluation gathered during 

the survey illustrate the product users' focus on evaluating activities 

overall, rather than on assessing the impact of any one set of materials. 

Example: LEA Level * The WEEA^ materials are used as a 
resource for planning a career guidance program for stu- 
dents at the secondary level. One workshop session of 
this, program is devoted to sex equity. The overall pro- 
gram is evaluated at its conclusion by administering a 
career development and interest inventory. No data are 
collected on the sex-fairness aspect of the program, but 
staff have reacted favorably to the materials and feel 
they have contributed to the success of the program. 

Example: SEA Level . The state education agency office 
uses WEEAP materials regularly, along with other materi- ~* 
als, to conduct LEA workshops. The goal is to develop an 
awareness of what sex equity is. Workshop participants 
complete evaluation forms at the end of the sessions, and 
SEA staff refer to these evaluations to determine which 
materials have been best, received. The WEEAP materials 
have been well received. 

Example: I HE Level . The WEEAP materials are used by the 
instructor in/ an introductory course for community college s 
student 8 exploring teaching as a career. One small part 
• of the course is devoted to the problem of sex discrimina- 
tion in educational agency hiring and promotion practices. 
The WEEAP materials were adapted for use in the classroom 
and were also pliced on a list of suggested readings to be 
used as resources for course papers. The sex equity 
aspect of the course is not covered in the examinations, 
but the instructor has the impression from student inter- 
est during class that the material has a positive impact. 

Example: Other . A "displaced hcmemaker H program coordi- *y 
nator referred to the WEEAP materials to confirm that her 
program was "on track. H Parts of the materials were used 
in workshops, along with other materials. The WEEAP 
materials did not suggest any changes in the program, 
which is already considered to b$ successful, an, impres- 
sion based largely on followup data collected by program 
c staff on women who have passed through the program. 

Were there any "bonus" outcomes from use of the materials? Twelve of * 
the 18 users, of WEEAP materials reported no unexpected or "bonus" outcomes 
from using the materials. Howeyer, three purchasers reported that after 
they had used the materials, other instructors ia their schools came to them 
to express interest in trying out similar materials or approaches.- One of 
these same purchasers said that her college was considering starting a 



women's studies program on the strength of the successful course she*haa 
Qffered uiing the WEEAP materials. She had. also had ma\y requests- for 
presentations from student and community groups* Other purchasers noted 
that use of the materials had helped develop trust in educational equity 
goals among those involved in product utilization, that teacher attitudes 
had changed through theit exposure to WEEAP classroom materials, and that 
the matej$als turned out to be unusually well suited for use with the 
university program staff, although they had originally been purchased for 
~ other purposes. ^ 

r - x 

Product Quality 

What was the technical and substantive quality of the materials pur- 
chased? Purchasers in this sample were generally very favorably impressed 
with the quality of the WEEAP products. Of the 20 interviewees who^ could 
comment on quality, 17 said* both technical and Substantive quality were 
either "excellent" or "goodi" while three said the quality was "OK." 

Among the favorable comments on technical quality were that prices were 
reasonable; the materials were easy to reproduce, nicely bound, on good „ 
quality paper, and the availability of rental audio-visual materials was_ 
helpful. .In terms of contend, users said the materials were well organized, 
well written, non-technical, self-contained, realistic, and fair to both 

sexes* 

Despite the general satisfaction with the materials, some negative 
comments were made, such as materials being cumbersome because of size or 
format. A preference was stated for typeset text over typed text. Some 
materials were considered to. be, biased to the secondary (rather than post- 
secondary) level. Other suggestions for improvement included better index-r 
ing, more use of simple charts and graphs, inclusion of updated statistics, 
and improvement of ethnic balance in some audiovisual materials. 

Future Directions 

\ What plans are, there for continued use of WEEAP products? Most pur- 
chasers of » WEEAP products plan to continue using them- Of the 25 purchasers 
interviewed, including users and non-users, 20 plan to use the materials in 
the future. Of the 18 who have already used the materials, 9 will maintain 

their current, level of use, 4 hope t>6 expand' their current use, and 5 will 

/ ■* 
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reduce or discontinue use of this particular product* Since no users 
expressed dissatisfaction with the WEEAP products', ve inquired into the 
reasons for reducing or stopping use* Four of the five interviewees would 
not use the products in the future because their funded project or course 

i 

had been terminated; one had found other WEEAP products more suited to the 
agency's needs* 

The reasons given by nonrusers for continued non-us£ (4/7) were as 
follows: ' 

• purchaser left materials at agency whore she is no longer 
employed, and the agenfcy lacks funding to use them; 

» , 

• materials have been misplaced or lost; 

• financial and community support are lacking; and 

• personal business interests have changed* 

What kind of educational equity m aterials should WEEAP develop in the 
future? This sample of product purchasers suggested a diverse array of 
materials they would like to see developed under WEEAP grants* Most fre- 
quently mentioned were materials addressing the changing roles and needs of 
men as well as women, and materials on non-traditional career choice; both 
types of material were mentioned by 6 of the 25 interviewees* The target 
groups most often mentioned as needing additional materials were older women 
and handicapped and minority groups, such as American Indians, Hispanlcs, 
Blacks, and "white ethnic women* M Table 9 presents the suggestions for 
needed WEEAP products and the target audiences to be served* Many of the 
areas and groups are already served by existing WEEAP products, but the 
interviewees either were unaware of these materials or felt that additional 
materials were needed* 

Is WEEAP' s current publication and dissemination system satisfactory ? 
During the interviews, we briefly 4^dcribed to the purchasers WEEAP' s pres- 

T 

ent" system for publishing and disseminating the materials developed* We 
then asked the purchasers for their opinions on the system, based on their 
own experiences with WEEAP and the Publishing Center* Of those purchasers 
vh^ qould comment on the system (18/25), most were generally satisfied with 
it (^/25). Six purchasers felt the system was not entirely satisfactory 
because 'it did not stiff iciently publicize the materials* They would like to 
see WEEAP or the Publishing Center do even more to advertise these products, 
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- -— Tablt 9 

WEEAP PRODUCT PURCHASERS 4 PREFERENCES FOR ADDITIONAL MATERIALS 

NOTE: Numbers refer to frequency with wftich the suggestion was mentioned in the 25 
interviews ♦ Double counting is due to multiple preferences of interviewees. 
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e.g., take part in more conferences and workshops nationwide, use a wider 
variety of channels for publicizing WEEAP materials, and so forth- 



User Comments^ on Dissemination. of WEEAP Products via Its 

Publishing Ce&ter: \ 

The system is a good one, especially the aspect of having 
a centralized location with a good screening system, 
t 

The descriptive catalog is very goo4 and the prices are 
reasonable* "* w 

Tne WEEAP Publishing Center is efficient and certainly 
better than commercial publishers or the Government 
Printing Office would be. 

The system would be improved if the Publishing Center 
worked more closely with the office in each state's edu- 
cational agency that has the most active contacts with » 
school districts. 

The Publishing Center is successful in reaching school- 
based educators, but some method is needed to bring 
independent purchasers into the system.' 



What suggestions were made for increasing public awareness of WEEAP 
products? The purchasers in this sample felt that WEEAP 1 s dissemination 
activities could be improved in two main ways: (1) by providing more oppor- 
tunities for potential purchasers to examine the materials first hand, and 
(2) by widening the distribution of printed publicity from th$ WEEAP Pub- 
lishing Center. Even the users who are pleased with the current publication 
and dissemination system said that it would be more effective if there were 
more presentations by WEEAP or Publishing Center staff at educators 1 confer- 
ences, workshops sponsored jo\ntly by WEEAP or the Publishing Center and 
groups such as the Sex and Race Desegregation Assistance Centers, Council on 
Interracial Books for Children, the American Personnel and Guidance Associa- 
tion, State Title IX agencies, and the National Diffusion Network. Sugges- 
tions for improved publicity included providing free sets of WEEAP materials 
to the equity-oriented state agencies to whom LEAs frequently turn for 
assistance, and mailing catalogs to school libraries and to agencies and 



individuals listed in directories of public and private service agencies. 
It was also suggested that the WEEAp/ Publishing Center advertise more in 
! publications — not only in professioi(al journals for educators, but also in 
^newsletters and magazines aimed at a) broader audience, such as social ser- 
vice and career planning agencies. / 
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Summary 



Results of the user survey indicated that WEEAP products were being 

y ♦ 

used by a diverse group of people for a variety of purposes related to the 
promotion of educational and career equity. 

WEEAP materials were most commonly used by adapting them to fit into an 
ot«oing course ov program. The WEEAP product might form the core of a course 
or workshop, but more often it was one of a^ number of materials used that 
were either developed locally or acquired from some otiier source. WEEAP 
i^aterlals were frequently seen as providing a reliable source of background 
information. They were often made accessible to a wide audience of users 
through libraries or resource centers. 

This sample of purchasers found the WEEAP materials to be of high 
quality and easfy to implement;. Nearly all said that the materials were 
effective in advancing educational equity and in helping users meet local 
heeds. However, few objective data were available to attest to the per- 
ceived impact of the WEBMP pro&cts. The evaluation data that existed or 
that could be compiled 4pre sketchy and did not focus on the -effectiveness 
of the WEEAP materials. in and of themselves. This was partly because few of 
the programs discussed were systematically evaluated and partly because 
WEEAP products were seldom used in isolation. 

When we compared our^findings on how* and with whom WEEAP products were 
being used and on the effectiveness of the products, we found a close corre- 
spondence between the experience .of this sample of purchasers and WEEAP's 
intended program outcomes. All the intended outcomes listed in boxes 31-36 
in the Detailed Logic Model ofr the Intended WEEA Program (shown in the fold- 
out figure following page<4) were included among the outcomes experienced 
by this sample of product, users. 
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There was general satisfaction among this group of purchasers with 
WEEAP f 8 current publication and dissemination system, although it was widely 
suggested that more opportunities be offerred to examine the materials 
firsthand* 

-4- 

Nearly all of those interviewed planned to continue their involvement 
in equity activities and to use WEEAP materials in the future* .Those who 
were uncertain or who did not, plan to continue were mainly stopping because 
of external reasons, such as lack of funding* Finally, this sample of pur- 
chasers planned to take a diverse approach to addressing equity issues and 
expressed a need for WEEAP products commensurate with those intentions* 
Materials related to equity in career choice will be needed by one-fourth of 
the interviewees, and the same number intended to expand their current 
efforts to emphasize human equity and equal educational opportunities for 
both men and women* 

In conclusion, thfe results indicated that this sample of product users 
achieved the short term impact intended by WEEAP, as represented in the 
detailed logic model of the program in Chapter I of this report* . ' 

Implications for WEEAP* s Future Evaluation Planning 

From this^small user survey we learned that any future large-scale 
survey of WEEAP product users should be planned to balance the following 
considerations: 

• effort required to locate and contact product users; and 

• usefulness of user responses as evidence of program impact 
and as a basis for program management decisions* . 

Effort Required to Locate and Contact Product Users 

It was much easier to obtain telephone numbers for .the 28 agencies in 
our survey sample than it was to schedule and complete the interviews* On 
the average, four telephone calls were needed to complete each interview, 
• and in one case, eight, preliminary calls were made before the actual inter-' 
view call could be made* Once appointments were made, the interviews were 
conducted with no problems* Respondents were invariably cordial and willing 
to share their views* Should a large-scale user survey be conducted, it 

-32- 
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would probably be advisable to start by "selecting a much larger sample than 
actually needed, and then limiting preliminary contact for the interview to 
two telephone calls. It" would also be advisable to limit the interview ques- 
tions to those found during bur^ survey to yield the most useful responses* 

r 

Usefulness of Responses ' 

In our user survey, some avenues of inquiry were more productive than 
others* The least productive area of ^inquiry related to evidence of product 
impact. We had originally hoped .that (1) the use^ survey would provide sev- 
eral instances of extensive product use for which evaluation data would be 

available, and that (2) these examples would make interesting short case 

■ 

studies Qf product utilization and inipact. However, it turned out that WEEAP 
materials , were most often ajiapj:ed to~fit into ongoing programs^and were used 
in combination with other materials. Thus, there was no "program" or 
"intervention" identifiable as exclusively WEEAP-developed. We also failed 
to find readily available data providing evidence of impact that could be 
reliably attributed to WEEAP product use alone with any degree of confidence. 

The most productive aspects of the survey were that it provided an, 
indication of (1) the amount of "product use, (2) the purposes for which the 
materials are being used,^('3> f the level of user satisfaction with the 
materials, and (4) the extent to whifch product use addresses outcomes of 
most interest to WEEAP. A The tfser survey questions related to "future direc- 
tions" also were useful, in that they served as a rough assessment of needs 
fot materials in certain areas "and for certain target groups* A larger-scale 
survey in the future could include a jieeds assessment that would help WEEAP 
to plan future priorities for product development. 

In analyzing the results of the user survey** the fo\jlowlnfc additional 
f evaluation questions arose that might be addressed in a subsequent evalua- 
tion: " * 

• Is there any significance to the small percentage of product 
users in 'our sample of Northeaster^ states? (Chapter III 
notes that states in this region accounted for a large part 
of the purchases ffco* tJhe WEEAP Publishing Center.) 

^ • What is the pattern in repeat purchases of WEEAP products 
(types of purchasers who repeat; types of products they 
purchase whan repeating)? What is the significance of 
repeat purchases? Why do purciiasers buy again? 

, X 
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• Are marketing efforts being directed at groups that repre- 
sent the most likely. purchasers! as evidenced by past pur-* 
chasing patterns? . ' 



Costliness of the User Survey ' 

The results of our small user survey suggest that this strategy can 9 
provide WEEAP management, with important quantitative and qualitative infor- 
mation pertinent to making decisions about areas of interest for product 
development, dissemination, and utilization. However, the cost per respon- 
dent of obtaining such comprehensive information was high. 

It is our judgment that WEEAP should consider less costly ways of 
obtaining similarly comprehensive data to meet its specific information 
needs* For example, the questions and responses from our small survey could 
be selectively adapted to develop forced-choice questions for a mail survey 
Qf users. The Publishing Center order form does include a short question- 
naire for purchasers, but It is reportedly seldom completed. As an alterna- 
tive, the Publishing Center could be funded by WEEAP to conduct a followup 
survey on product use, to be mailed out ten months to a year after the order 
is shipped. 
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Chapter III 

fr ANALYSIS OF WEEAP PRODUCT SALES 



Purpose 



While the WEEAP Publishing Center* s monthly sales records have provided 
a continuing record of what products sell best, no systematic, comprehensive 
analyses of related factors or sales trends has been done. ; The analyses we 
carried out were intended to provide very rough estimates of answers to 
several questions about WEEAP product dissemination r* 

Purchasers and What They Buy 

• What types of items were purchased the most? 

• What types of purchasers bought the most? 14 
Did certain types of purchasers buy certain types of items? 



Regional Variations 

' • Were there regional variations in the amount of purchases 
made? 

• Were there regional variations in the types of items pur- 
chased? 

• Were there regional variations in the types of purchasers? 
The Cost Factor 

» 

• How much did most WEEAP items cost? 

• What priced items sold the most?* 

• Did price influence the types of items purchased? 
Did price influence the types of purchasers? 



* It was not within the scope of the rapid feedback evaluation to consider 
the availability or sales performance of sex-equity materials developed 
and marketed by noxr-WEEAP sources; 



Procedure 

Sources of Data 

Two types>f existing records provided the data base for our analyses: 

• 388 purchase invoices representing a 5% random sample o.f all 
invoices on file at, the WEKAT Publishing Center. The 388 
invoices represent 2,488 items sold and $18,130 in dollar 
volume* 

• Summary sales records for every item sold as of the end of 
1978, 1979,< and 1980, representing a total dollar volume of 
$333,866. 

Although there were 184 items in the WEEAP Publishing Center catalog and 
"new release" flyer, there wera actually 193 different items sold. The nine, 
additional items (those not listed in the catalog) were subparts of "kits" 
for which we had no record of price. Of the 193 different items (including 
the kit subparts) available, we have sales data for 140 items. The missing 
sales data can be accounted for in the following manner: 

• 32 itjems listed in the catalog frere audiovisual rental items. 

• 5 "new release" items had not yet sold by December 1980. 

» The remaining items were sold in the late fall of 1980 and 
had not yet been incorporated into the Publishing Center's 
sales records. w 

Thus, the summary sales records include: 

• 193 items sold 

• 184 items with price data 

• 140 items with sales data 

Coding, of Data 

The summary' sales records of the 18£ items in the WEEAP Publishing 
Center's catalog and "new release" flyer were hand coded to provide the 
following information: ^ 

• Date the .item became available for purchase 

• Type of item 

• Price* of item 



^ • Number sold each month 

• Number sold each year 

The 388 Invoices, were also hand coded to providfe the following inf ondation: 
e . Type "of item 

e Type of purchaser 

e Price of item 

• State/region of purchaser 

In coding "type of item," we adhered to the product's placement in the 
WEEAP Publishing Center catalog* The catalog sorts mat' ,al into six major 
categories according to" the audience and use the material was primarily 
intended- for : 

# InBervice and Staff Development materials include products 
J^wftich focus on any of the following: staff training 'in sex 

equity policies and practices; broadening educators 1 aware- 
ness of sex-role expectations, . biases , and perceptions; 
training educators to evaluate curriculum materials with 
regard to sex typing; providing information on planning and 
evaluating school or community based sex equity programs* 

•* Curriculum Materia ls are intended for school or community 
based use with students* All these materials address sex 
equity concerns* ' Most of the materials are oriented to a 
particular topic or field of study, e*g* , .physical educa- 
tion, math aftd science, language arts* . 

*\ 

# Counseling and Guidance materials are oriented to both edu- 
cators and students* Some of these materials consist of 
curricula which focus on expanding students' awareness of 
sex-role values, attitudes, and biases; other items in this 
category are f cr students to use in self assessment and 
career exploration* This category also includes manuals and 
information for counselors who want to set up a counseling 
program for women or girls* 

♦ . Career Development materials fo ( ;us on career concerns and 

r options*. The materials are intended for students, parents, 

and educators and generally address traditional and non- 
traditional career options in particular fields, e*g*, voca- 
tional-technical, math and science, continuing education* 
Some of the materials include manuals for planning career 
development programs* 

• Early Childhood Education materials are Intended for use , 
with young children by educators And/or parents. These 
materials promote awareness of how sex typing limits a 
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child's development and emphasize the importance of foster- 
ing non-sexist environments for young children. 

• Educational Administration materials include products deal-* 
ing with the role of women in educational administration and 
encouraging more women to pursue careers in this area. 

Some products appear more than once in the WEEAP Publishing Center 
catalog because they fit into more than one category. However, when we . 
coded these materials, we used the item-type classification under which the 
particular item was first located in the catalog. There were also several 
items that were developed after the WEEAP materials catalog was printed. 
These items were advertised on a supplemental form and were classified as 
"Ifew Titles." ' 

Each of the six major types of items in the WEEAP catalog is broken 
down further into subcategories. For example, a career development product 
focusing on the math and science area is classified under "Career Develop- 
ment** and further classified as "Math and Science." We therefore coded 
items according. .to 27 different categories (including major categories and 
subcategories). Whenever an item was subclassified, it retained its major 
classification and was coded twice. "Consequently, while the detailed 
27-categbry system provided considerable detail, it included a great_deal of 
multiple coding* In presenting results, we often elect to use the simpli- 
fied major categories in which all items are counted only once. (When we 
compared tables based on single and multiple category codes, we found only 
minimal, differences in overall trends and patterns.) 

i 

Data Analyses and Generalizabllity of Findings 

Several descriptive analyses were performed with these data. In most 
cases, ^counts of total iten^s sold and counts of total dollar volume id sales 
were examined. We also examined average monthly sales figures for each 
item. When analyzing sales patterns, we controlled for both item avail- 
ability and item price. * 

The following caveat should be applied in generalizing from findings 
based on the random, sample of 388 purchase invoices. We can be 95% sure 



* Although we examined item sales for 1978, 1979, and 1980, we used the 
prices based on the current (1981) .catalog prices which may or may not be 
identical to prices in previous years. 



that the percentage in tables based' on this sample will be within about five 
percentage points of the true value for the total population of purchasers. 
(One purchase invoice represents one purchaser, but may represent one or , • 

many purchases of individual items in the WEEAP Publishing Sinter catalog.) 

% Thus, if we found that the proportion of sales to elementary /secondary 
institutions, based on our sample of invoices/purcha sers , was 31% of the . ^ 
total volume of sales, we can be relatively sure that between 26% and 36% of 

*all item sales were made to purchasers in this category. Whenever possible, 
however, we present two tables*— t>ne based on the three-year sales records 
provided by the WEEAP Publishing Center for 1978, 1979, and 1980, and one 
based on the random sample. Then the reader can see how close the, cofrre- 
sponfdeftce is, and. this provides an important extra measure of certainty. 
For example, the reader will find that the three-year sales records, showed 
Staff Development items accounting for 41% of all item sales through 1980 
and that the 5% random sample of invoices (drawn from all those on file ap 
of the end of the first third of 1981) showed a roughly comparable 39%. 
(See Tables 12 and 13, respectivaly*-)- 
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Results 

Results of the data analyses are presented in the following three major 
sections: t 

# Purchasers and What They Buy 

• Regional Variations 

* The Cost Factor 

Each section addresses the relevant questions listed at the outset of this 
chapter. 

Purchasers and What They Buy 

What types of items were purchased the most ? To provide a general 
frame of reference, Table 10 presents the dollar volume and number of items 
sold by the WEEAP Publishing Center* for 1978, 1979, and 1980. 
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Table s 10 

• ' OVERALL^ SALES ACTIVITY -OF THE WEEAP PUBLISHING CENTER 

BASED ON YEARLY SAXES RECORDS 
■ » 

Number of Items Sold Dollar Volume 

N . 2 Z I * 

1978 . 1,746 ( 42)L $ 30,493 ( 9%) 

1979 - . 12,112 (282) $119,117 (362) 
\ 1980j ' 28,940 (662) ~ $184,256 (55Z) ' 

TOTAL 42,798 (982) - $333,866 (1002) 

Based on the total number of items sold (42,798 items) during the three-year 
period from 1978 through 1980, 90% of the sales were of items classified as 
Staff Development, Curriculum, and Career Development. Table 11 presents 
the sales data* for each year. This table indicates tha*f the top-selling 
item types are generally stable across, the three-year period.* 

Table 11 

MOST FREQUENTLY PURCHASED ITEMS BASED ON YEARLY SALES RECORDS 

Number of Number of Number of 

Items Sold Items Sold Items Sold Total Sold 
Item Type 1978 1979 1980 ' (1978-1980) 







2 ' 


N 


2 . 


N 


2 


N 


2 


Staff Devel. 


1,043 


59.7X 


6,207 


512 


10,119 


352 


17,369 


40.62 


Curriculum 


* 5 


.3X 


1,644 


142 


9,080 


312 


10. 729 


25.12 


Career Devel. 


619 


35.52 


3,007 


252 


6,874 


242 


10, ^00 


24.52 


Guidance , 


' ^ 79 


4; 52 


983 


82 


2,040 


72 


3,102 


7.22 


Ed. Ada. 


0 


0.0% 


' • 271 


22 


576 


22 


847 


2.02 


Early Child. 
Education 


0 


0.02 


•> 

0 


02 


240 


12 


240 


0.62 


New Titles 


0 


0.02 


0 


_02 


11 


(nil) 


22 


0.12 


TOTAL 


1,746 


100.02 


12,112 


1002 


28,940 


1002 


42,798 


100.12 



* The reader is reminded that W2EAP did not- establish specific priorities 
until the 1980 fiscal year. Our findings are based on sales records for 
products developed under grants funded, prior to that time, when there had 
been an. implicit emphasis on funding the development of # training and cur- 
riculum materials. " , ~ * 
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As Table 12 shows, a similar pattern characterized the 5% random sampl 
of invoices (N - 388 invoices, 2,488 items, and $18,130 in dollar volume). 

Table 1.2 



MOST FREQUENTLY PURCHASED ITEMS 
BASED ON A 5% RANDOM SAMPLE OF INVOICES (N-388) 



Item Type 


Number of 


Items Sold 


Dollar- 


Volume 


f 


N 


X ' 


i 


•r 

M 


Staff Development 


964 


38. 7% 


t 9,079 


, 50. OX 


Curriculum » 


621 


25.0% 


$ 4,420 


24.4% 


Career Development 


607 


24. 4% 


$ 3,165 


17. 5X 


Guidance 


12 6 


5.1X 


$ 892 


4.9% 


Ed. Administration 


87 


3.5% 


$ 287 


1.6% 


Early Child.' 'Education 


52 


2.1%" 


$ 169 


0.9% 


/■ New Titles* . 


31 


1.2% ■ 


$ 118 


0.7X 


TOTAL 


2,485**- 


100.0% 


$18,130' 


100.03; 



To determine whether the high rat^ of sales of these item types was 
influenced by their* availability, we first considered availability for each 
year for each type of item (Tabl* 13). As can be seen in Table 13, there , 
are more items available in Staff Development, Curriculum, and Career Devel 
opment for each year. 

Table 13 

TYPES OF ITEMS AVAILABLE OVER THE THREE-YEAR PERIOD 1978-1980 



1978 1979 1980 



Item Types 


N 


X 


N 


X 


N 


X 


Staff Development 


• 10 


42% 


34 


4 93! 


58 


41.4% 


Curriculum 


5 


213! 


22 


323! 


48 


34.3% 


Career Development 


7 


293! 


■ 9 


13% 


21 


15.0% 


Guidance 


2 


8% 


3 


4% • 


6 


4.3% 


Early Child. Education 


0 


• 0 


0 


0 


1 


0.7% 


Ed. Administration 


0 


0 


1 


. 13! 


3 


2.1% 


New Titles 


JO 


0 


,_0 


_0 


3 


2.1% 


TOTAL 


24 


100% 


69 


■ 99% 


. 140 


99.9% 



(fete: Table based on only those items for which we had sales data.) 
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We next- Examined each item type, the number of items available in that 

category, and the average monthly sales for each item. As Table 14 shows, 

f 

those item types with a large number of items available had lower average 
monthly sales per item. Thus, the Staff Development category, which had 
available 58 items from which a purchaser could choose,* had a monthly avei 
age of 19 sales per item . This can be compared to the Early Childhood 
Education category, which had only one item from which a purchaser could 
choose and a monthly average or 48 sales per item . 

Table 14 



AVERAGE MONTHLY SALES PER ITEM BY ITEM TYPE 
BASED ON SALES RECORDS 



Item Type 



Number of Items 



Average Monthly Sales. 





N 


% 




Staff Development 


58 


41.4% 


19 


Curriculum 


48 


3 4.. 3% 


24 


Career Development ' 


21 


15.0% 


39 


Guidance 


6 


4.3% 


40 


Ed. Administration 


3 


2.1% 


32 


Early Child. Education 


1 


0.7% 


48 


New Titles 


3 


2.1% 


4 


TOTAL 


' 140 


99.9% 





* Table includes only those items for which we had sales data. 

** The pattern was essentially the same for 1978 and 1979. Note the gener- 
ally inverse relationship between the number of items available for sale 
and the average monthly sales per item for that type of item. 

We reason that the explanation for the inverse relationship between the 
number .of items available for sale in a particular category and the average 
monthly sales per item for that category is as follows. In effect, materi- 
als in the Early Childhood Education category have a higher average monthly 
sales per item than items in the Staff Development category because a pur- 
chaser wishing to purchase the former had virtually nothing to choose from. 
The purchaser either bought the one item available or did not. On the other 
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hand, a potential buyer could choose among 58 different items in the Staff 
Development category. Thus, the average monthly sales per item would be 
lower for Staff Development materials and for all item types with a large 
availability * 

To explore further the issue of item availability and item sales, we 
compared the percentage of items sold "as of the end of 1980 and the per- 
centage available • Table 15 shows that for some types of items, the -per- 
centage sold was larger than the percentage of items available. For other 

Table 15 

COMPARISON OF ITEMS SOLD AND ITEMS AVAILABLE 
AS 0? THE END OF 1980 



Number and Percentage 
of Items Sold (1980) 



Number and Percentage 
of Items Available (1980) 





N* 


% 




% 


Staff Development 


17,36$ 


40.6% 


58 


41.4% 


Curriculum 


10,729 


25.1% 


48 


34.3% 


Career Development 


10,500 


24.5% 


21 


15.0% 


Guidance 


3,102 


7.2% 


6 


4.3% 


Ed* Administration 


847 


2.0% 


3 


2.1% 


Early Child. Education 


240 


0.6% 


1 


^ 0.7% 


New Titles 


11 


(nil) 


3 


X2.1% 


TOTAL 


42,798 


100.0% 


140 


99^9% 



* Table includes only those items for wliich sales information was available. 



types of items, Che percentage sold was smaller than the percentage of items 
available. We believe there is a suggestion that purchasers may be desiring 
more of an item type if the percentage of items-aold is larger than the per- 
centage of items available. When the percentage sold is smaller than the 
percentage available, the implication may be that the "over-supply" point 
for this item type is approaching or has been reached. If this reasoning is 
correct, Table 15 indicates that there may be a need for more materials in the 
Career Development and Guidance categories. In contrast, additions to the 
Curriculum category should be made very selectively, as the "over-supply" 
point may have been reached — at least for the particular kinds of curriculum 



materials being offered. It also appears that Staff Development materials 
are adequately meeting purchaser demands* ' 

What types of purchasers bought* the most? Based on our 5% random sample 
of invoices, we found that approximately 87% of the purchasers (2,154 pur- 
chasers) were either postsecondary institutions, elementary/secondary insti- 
tutions, or SEAs. The same three purchaser types also provided 90% of the 
dollar volume, with slight variation between the number of items sold and 
dollar volume as shown in Table 16. The WEEAP Publishing Center' s* marketing 
campaigns have emphasized these types of potential buyers because most 'of 
the materials available to date are highly appropriate for them. 



Table 16 

MOST FREQUENT PURCHASERS BASED ON A 5% RANDOM 
SAMPLE OF INVOICES (N-388) 



Purchaser Type 


Items 


Sold 




Dollar 


Volume 


N 


% 




£ " 


Z 


Postsecondary 


803 


32.3% 


$ 


3,891 


21.5% 


Elementary/Secondary 


769 


30. 9% 


* 


5,956 


32.9% 


SEA* _ 


587 


23. 6% 


$ 


6,537 


36.1% 


Individuals 


81 


3.3% 


* 


393 


2.2% 


Women's Organizations 


80 


3.2% 


* 


266 


" 1.5% 


Businesses 


70 


2.8% 


$ 


419 


2.3% 


Local Agencies 


62 


2.5%' 


$ 


335 


1.8% 


Professional Organizations 


27 


1.1% 


$ 


293 


1.6% 


Federal Agencies 


6 


0.2% 


$ 


33 


0.2% 


State Agencies* 


3 


0.1% 


$ 


10 


(nil) 


TOTAL 


2,488 


100.0% 


$18,130 


100.1% 



State Education Agency (SEA) purchasers and other state agency purchasers 
were coded separately in our analyses. 



Did certain types of purchasers buy certain types of items ? Although 
all types of purchasers bought a wide variety of items, most purchasers 
bought the " top-selling" types of items — those categorized as Staff Devel- 
opment, Curriculum, and Career Development. Based on the random sample of 
388 invoices and uping 27 categories to classify types of items, we found 



-44- 



further that: elementary/secondary institutions were the purchasers of 75%' 
of the items (15 items) sold in the subcategory Galled Career Development 
Curriculum , .representing 68% of the dollar volume in sales for this type of 
item. The remaining 25% (9 items) were bought by SEAs and professional 
organizations. Not surprisingly, in the Conmunity and Junior College Career 
Development subcategory, 78% (22 items) of the items sold, representing 79% 
of the dollar volume for this type of item, were purchased by postsecondary 
institutions. Similarly, 61% (222 items) of the items sold in the C ontinu- 
ing Education Career Development subcategory, representing 58% of the. dollar 
volume for this type of item, were purchased by postsecondary institutions. 

Sixty-one percent (213 items) of language arts curriculum materials, 
representing 55% of the dollar volume in sales for this subcategory, and 52% 
(59 items) of inservicc career development materials, representing 67% of 
the dollar jrolume in this subcategdry, were purchased by elementary/secon- 
dary institutions. 

SEA purchasers accounted for 55% (107 'items) of both the dollar volume 
and the dumber of items sold in the Vocational-Technical Career Development 
subcategory. 

• Thus, analyses of sales in subc ategories (asopposed to major cate- 
gories) provides a useful and comprehensive picture of the preferences of 
particular types of buyers for particular types of products. 

Regional Variations 

Were there regional variations in the amour t of purchases made? For 

this analysis, we used the Education Department's regional boundaries which 

i ncilude the following 10 regions: 

Region 1: Maine, Massachusetts, New Hampshire, Vermont, Connecticut, < 
* Rhode Island 

Region 2: New York, New Jersey, Puerto Rico, Virgin Islands 

Region 3: Delaware, Maryland, Pennsylvania, Virginia, West Virginia, 
District of Columbia 

Region 4: Alabama, Florida, Georgia, Kentucky, Mississippi, North 
Carolina, South Carolina, Tennessee 

Region 5: Illinois, Indiana, Michigan, Minnesota, Ohio, Wisconsin 

Region 6: Arkansas, Louisiana, New Mexico, Oklahoma, Texas 
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Region 7: Kansas, Nebraska, Iowa, Missouri 

Region 8: Colorado, Montana, North Dakota, South Dakota, .Utah, Wyoming 

Region 9: Arizona, California, Hawaii, Nevada 

Region 10: Alaska, Idaho, Oregon, Washington. 



As can be seen in Table 17,< which is based on the sample of 388 
invoices, both the number of items sold and the dollar volume indicated that 
Regions 1 and 3 were the largest purchasers, together representing 38% of 
the sales in our sample. 'In contrast, in this sample, Regions 6, 8, and 10 
together accounted foi^ 10% of product sales. Recalling the caveat on gener- 
alizing findings from our invoice sample (see end of Procedure section in * 
this chapter), the true percentage of sales to Regions 1 and 3. combined 
would fall somewhere between about 33% and 43% while the true percentage of 

Table 17 

REGIONAL VARIATIONS IN AMOUNT OF PURCHASING 
BASED ON A 5% RA>ND0M SAMPLE OF INVOICES (N-388) 



1. 


> 

ME, 


Region 
MA, NH, 


VT, 


CT, 


RI 


. % of U.S. 
Population* 

. 5% 


Quantity of 
Items Sold 
N , % 
629 - 25.2% 


$ 


Dollar Volume 
$ • % 
57022 2777% 


2. 


NY, 


NJ, 


PR, 


VI 






12% 


201" 


8.1Z 


. $ 


1,065 


5.9% 


3. 


DE, MD, 


PA, 


VA, 


WV, 


DC 


112 - 

1 


336 


13. 5Z 




3,560 


,19.6% 


4. 


AL, 
NC, 


FL, 
SC, 


GA, 
TN 


KY, 


MS, 




17% 


278 


11.2% 


* 


2,536 


c 

14.0% 


5. 


IL, 


IN, 


MI, 


MN, 


OH, 


WI 


20% 


321 


12. 9Z 




1,582 


8.7% 


6. 


AR, 


LA, 


NM, 


OK, 


TX 




11% 


40 


1.6% 




272 


1.5% 


7. 


KS, 


NE, 


IA, 


MO 






' 5% 


299 

s 


12.0% 




1,854 


10.2% 


8. 


CO, 


MT, 


ND, 


SD, 


UT, 


WY 


3% 


76 


3.1% 




384 


2.1% 


9. 


AZ, 


CA, 


HI, 


NV 






12% 


185 


7.4% 




1,228 


6.8% 


10. 


AK, 


ID, 


OR, 


WA 






4% 


123 


4.9% 




627 


3.5% 


TOTAL 












100% 


2,488 


99.9% 


$18,130 


100.0% 



*Based on 1980 U.S. census data* 
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sales to Regions 6, 8> and lp together would fall somewhere between about 5% 

— p 

and 15%. This "confidence zone" t : s only approximate. The actual zone of 
confidence is affected by the extent to which the number of items sold and 
the. dollar volume for sales we report for a particular region or group of 
regions includes a high propprtion of "large" invoices as compared to 
"small* invoices. What is of interest here is not the numbers per se , but 
the apparent imbalance in regional* buying patterns. The combined figures 
for populations in Regions 1 and 3^and for populations in Regions 6, 8, and 
10 are similar— 16* and A 18%, respectively. (Population figures taken from 
1980 U.S* census data.) If it is fair to hypothesize that, bas^d on U.S. 
population figures, the potential market for WEEAP products in these regions 

i .> ^ . * * 

is more similar than suggested by the sales figures, then it will be appro- 

o 

priate to investigate further the reasons for the'apparent imbalance and to 

■r • 
' e 

determine whether implications for marketing can be identified. 

* * * * 
Were there regional variations in^the types of items purchased ? Even 

•^though in our 5% random, .sample of invoice's some regions purchased more than 
other regions, the pattern, in types , of ifcems purchased did not differ sub- 
stantially from analyses w<e presented earlier in this chapter. That is, 
all regions mostly bought items in the.Sta^f Development, Curriculum, and 
Career Development categories. This finding was consistent for both the 
number of items sold and for dollar -volume of sales. 

Were there regional variations in types of purchasers? Based on the 
random sample of 388 invoices, we found that there was some regional varia- 
tion in the type of purchaser associated with sales to a particular region. > 
For example, 68Z (169 purchas ers /invoice s ) „coded as "SEA" were from Region 
1; 67% (48 purchasers/ invoices) coded as "Businesses" were from Region 10. 
The impprtant point is tfcat wherte percentages are this la'rge, further inves- 
tigation of regional variations in types of purchasers and the reasons for 
them could assist WEE/? in planning future strategies for marketing products. 

The Co^t Factor 

How much did most WEtAP items cost ? We classified the 184 items for 
which we had prices according to the three cost categories* shown in Table 18. 



* All WEEAP products must be sold at cost according to the Publishing Center f s 
contract. 

er|c -4 7 -^ 
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Table 18 

NUMBER OF ITEMS IN THREE PRICE CATEGORIES* 
Price Category of Items Number of Items (N-184) s 

Inexpensive (less than $5) 79 / (43%) 

. Moderate ($5.01 - $9.99) 48 \(26%) 

Expensive ($10 or more) 57 (31%) 

TOTAL .184 (100%) 

* Table based on all items in the WEEAP catalog that h^ad price data. 

As indicated, although there ;are many items available in all three price 
categories, 43% of the items cost less than five dollars and more than 
two-thirda r Qi the' items cost less than ten dollars. " t 

Table 19 shows the number of items available each year in each price 
category; it includes only those items for* which, we had both sales and price 
data. The number of available items has increased each year; and the pro- 
portion of " inexpensive" items has always been higher than the proportion of. 
either "moderate" or "expensive" items. • However; the differential decreased 
each year as more expensive items (e.g., audiovisuals) were produced. 

Table 19 5 > ■ » 

NUMBER OF ITEMS IN EACH PRICE CATEGORY FOR 1978, 1979, and 1980* 







Number of Items 






Price Category of Items 


1978 


1979 


1980 


Inexpensive (Less than $5) 


16 (67%) 


40 (58%) 


72 


(51%) 


Moderate ($5.01 - $9.99) 


5 (21%) 


16 (23%) 


24 


(17%) 


Expensive ($10 or more) 


3 (12%) 


13 (19%) 


44 


(31%) 


TOTAL 


24 (100%) 


69 (100%) 


140 


(99%) 



* Table based on items for which we have both sales and price data. 

c 

What priced items sold the most? As shown in Table 20, which is based 
on overall sales records for all items 3old in 1978, 1979, and 1980, it 
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appears that expensive items sell less well than inexpensive or moderately 
priced, items. 

* • 

Table 20 

TOTAL ITEM SALES BY PRICE CATEGORY FOR THE • 
THREE-YEAR PERIOD 1978-1980 

Total Number of Items Sold 

Price Category of Item 1978-1980 

Inexpensive (Less than $5) 28,731 (70Z) 

Moderate (*5.01 * $9.99) 7,730 ' (19%) 

Expensive (fclO or more) A; 719 (11%) 

TOTAL , 41,180* "(100%) 

*This number conflicts with the total sales figure in Table 10. This 
discrepancy results from the fact that Table 20 represents those items 
for which we had both sales and price information/ In contrast, 
Table 20 includes all sales regardless of whether we had price data. 

Many of the expensive items are audiovisual materials (films or video- 
tapes). Of the 34 audiovisual materials available for purchase, all 34 are 
also available for rental at a considerably lower price. Consequently, . 
lower sales of expensive items may not indicate the lack of popularity of 
these items but rather the option to obtain many of them at a lower price 
through rental. 

Did price v influence the types of Items /purchased? Using the random 
sample of 3tf8 invoices, we examined sales data for the number of items sold 
and the dollar volume in sales for the various classifications of item 
types. The overall findings remained stable, regardless of item price, 
number of items sold, or dollar volume. In the inexpensive price range, 
items classified as Staff Development, Curriculum, and Career Development 
accounted for 85% (1,228 items) of the number of items sold and 84% ($3,936) 
of the dollar volume. For moderately priced items, these three item types 
accounted for 93% (534 items) of the number of items sold and 92% ($3,784) 
of the dollar volume. For expensive items, these three item types accounted 
for 87% (319 items) of the items sold and 95% ($3,936) of the dollar volume. 



Did price Influence the types of purchasers? For Inexpensive items, 
the post secondary institutions*, elementary /secondary institutions, and SEAs 
accounted for 84% (1,233 items) of the items s.old and 87Z ($4,018) of the * 
dollar volume. For moderately priced items, 'these three types of purchasers 
accounted for 87% (502 items) of both the number of items sold and dollar 
volume ($3,602). For inexpensive items, these three purchaser typ%s 
accounted fox 95Z (317 items) of both the number of items sold and dollar 
volum* ($8,766). Thus, the finding in our earlier analyses that these three 
types of purchasers accounted for most of the purchases of WEEAP products — 
whether by number of items bought or by dollar volume — held up within each 
of the three price categories. 



Summary 

Overall, the. production and sales of WEEAP products increased dra- 
matically during*' the three years on. which our analyses were based. The 
number of available products at the end of 1980 was double the number avail- 
able at the end of 1979 (140 as compared to 69, based on items for which we 
had sales data) . The number, of items sold more than doubled between 1979 
and 1980 (12,112 and 28,940 items sold, respectively). Cumulative sales 
records showed that as of the end of 1980, 42,798 items, representing 
$333,866 in dollar volume, had been sold. In 1980-81, the Publishing Center 
offered 193 WEEAP items for sale. 

The results shpwed that the WEEAP Publishing Center and the WEEAP prod- 
ucts themselves were succeeding in appealing to those segments of the educa- 
tional equity market for which the materials were intended. The largest 
purchasers were postsecondary institutions, elementary /secondary institu- 
tions, and SEAs. 

The most frequently purchased items were Staff Development, Curriculum, 
and Career Development materials. This finding was not surprising since 
most of the materials produced under WEEAP grants prior to 1980-81 apd 
approved for publication and dissemination were of these types. A comparison 
of the types of items sold with the types of items available for sale sug- 
gested that as of the end of 1980 a need for more materials in the Career 
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Development and Guidance categories might be indicated by the fact that the 
percentage ~sold was larger than the percentage available for items in these 
categories. On the other hand, the, "over-supply 11 point may be near or may 
have been reached for items in the Staff Development and Curriculum categories; 
Additions to these two categories should be made selectively to minimize 
competition between highly similar materials and to meet purchasers 1 prefer- 
ences for particular* kinds of materials 

While there ^were regional variations in the amount of purchases i buying 
patterns for item types were consistent; that is, the top-selling types of 
items in every region were Staff Development, Curriculum, and Career Devel- 
opment materials, there were a few regional differences in purchaser types, 
but in general, the purchasers from most regions tended to be either elemen- 
tary/secondary institutions, postsecondary institutions, or SEAs. 

Finally, we looked at item price and at purchasit*g patterns in relation 
to it. Overall, 51% of the WEEAP products being sold by the end of 19&6 
were inexpensive (less than five dollars) and more than two-thirds (68%) of 
them were price4 under ten dollars. While expensive items did not sell as* 
well as moderately priced tor inexpensive items, many of the expensive items 
were available to users i on an inexpensive rental basis. Regardless of 
price, the most frequently purchased items were Staff Development, Curricu- 
lum, arid Career Development materials, and the major purchasers remained 
postsecondary institutions, elementary/secondary institutions, and SEAs. 

Implications for WEEAP 's Future Evaluation Planning 

We conclude this chapter with a discussion of factors for WEEAP to con- 
sider in planning for future evaluations of product dissemination: 

• questions left unanswered by our sales analyses; 

• categories used to classify WEEAP products; 

• lack of computerized sales recofds; 

• qualitative information needed to illtpinate quantitative 
analyses. 



i 



ERIC 



Unanswered Questions 

Our data analysed left a number of questions unanswered* For example: 

• What motivates purchasers to buy products Intended for other 
types of purchasers? 

While our analyses show that the most substantial portion of 
sales of top-selling types of items may.be accounted for by the 
types of purchasers for whom these products were intended, there 
remains a modest proportion of sales to other types of buyers* 
WEEAP wants to increase the flexibility and adaptability of 
. their products for a wide range of users wherever that is 
feasible* < Information from buyers for whom the products were 
not originally intended could conceivably provide excellent 
guidance for future product development* v 

• What are present regional sales patterns and the implica- 
tions for future dissemination of WEEAP products? 

Our finding of regional difference^ in WEEAP 1 s product sales 
must be interpreted with the caveats applicable to our sampling 
procedure (explained in detail in conjunction with presenting 
these findings)* Furthermore, the sample on which our analysis 
was based included invoices dating back to 1978* It is pos- 
sible that the regional sales pattern for 1981 might differ and 
that the imbalances- we noted in our sample would not be present 
or would be much smaller or have a different configuration if 
1981 invoices were the basis for the analysis* However, if 
substantial regional variations K are found to exist at present, 
then additional questions can be addressed: Can regional vari- 
ations be attributed to differences in attitudes, resources, 
marketing strategies, or other factors or combination of fac- 
tors? Are such factors amenable to influence through strate- 
gies WEEAP can implement? Will the payoff be worth the cost of 
selectively, implementing additional marketing strategies? 

• What supply/demand gaps are there? 

Our data analysis only superficially addressed supply and 
demand ratios; we compared the percentage of items sold with 
the percentage of items available for sale within erch type of 
item in broad categories* We reasoned that the gaps noted 
might suggest that certain item types may be over- and others 
under-supplied. However, kur data were insufficient for a full 
assessment of the situation* For example, when sales of an 
item type were much less than the availability of that item 
type, we could not determine directly whether this meant that 
the various items available in that category were so similar 
they are competing with each other, or whether potential buyers 
found many of the 'items in that category less appealing than 
items available through other (e*g*, commercial) sources, or 
whether the items available in that category did not suit the 
situations in which many potential buyers intended to use them, 



-52- 

64 



or some other reason. 'If such oreasons were known, WEEAP f s 
future product development and publication efforts could be 
adjusted accordingly • A more complete assessment of the supply/ 
demand characteristics of WEEAP and non-WEEAP sex equity prod- 
ucts is warranted at this time because, with 193 items available, 
for sale and 90% of them being in three top-selling categories, 
WEEAP can afford to be particular about what products are 
developed and disseminated. 

* 

Categories Used to Classify WEEAP Products 

In codiijg the "type of item* for our sales analyses, we used the prod- 
uces classification in the WEEAP Publishing Center catalog. While those 
categories may meet WEEAP 1 s marketing purposes, they were less useful for 
our data analyses and limited the questions we could answer. 

For instance, categories used in-the catalog are not necessarily con- 
cordant. Some categories represent topical content, e.g., Early Childhood 
Education; other categories represent structural type, e.g., Curriculum. 
Lack of concordance contributed to our problem of non-exclusivity of cate- 
gories for data analyses. While we decided to use the category under which 
an item was first listed in the catalog for analyses based on "single" codes 
as opposed to "multiple- codes, we were not completely certain that this 
choice necessarily represented the best item "type" for a particular product 
Future evaluation planning might consider refining item classifications and 
accompaigring the classification scheme with clear criteria for deciding on 
the assignment of an item to a particular classification-. . 

Of course, the exact choice of categories to be used iit coding data 
for future sales analyses should be based on the specific questions to be 
addressed. For example, the classification scheme we used did not provide 
sufficient breakdowns of grade levels for which products were appropriate. 
It is true that many WEEAP products are suitable for use a,t both elementary 
and secondary levels, and this probably accounts for not breaking them out 
separately in the catalog. However, we could not answer questions about 
sales or supply /demand for items at . the "elementary" grade' level or the 
"secondary" grade level because the classification scheme we used did not 
separate them — it combined them as "elementary/secondary" items. - 

There may well be other classification refinements WEEAP will wa.;t to 
consider. For example, categories such as items for sale vs. items for 
rent, audiovisual materials vs. print materials, materials incorporating 
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lesson plans and many short activities vs. materials which do not lend them- 
selves to incorporating these features. Other categories might also be 
relevant to answer specific questions about WEEAP product development, dis- 
semination, and utilization in the future. 

Lack of Computerized Sales Records 

The sales analyses we reported in this chapter were a one-shot effort 
which WEEAP may decide not to replicate. Our major purpose in the rapid 
feedback evaluation was to assess the feasibility of doing such analyses if 
further evaluation of WEEAP product dissemination is deemed to be desirable 
in the future. Another important purpose was to provide program management 
with a very rough estimate of progress being made in meeting WEEAP 1 s dis- 
semination objectives. 

We knew that hand coding would be a major task, and that is why analy- 
ses that could not be based on the WEEAP Publishing Center's sales records 
for 1978, 1979, and 1980 were based instead on a 5% random sample of all' the 
invoices on file. Had a computerized billing system been provided for in 
WEEAP* s contract with its Publishing Center from its inception, the analyses 
could have been completed in a fraction of the time and could have been 
based on the total universe of item sales rather than a random sample of 
invoices, each invoice representing a different quantity and/or doliar 
amount in item sales. 

If the Department were to provide for a computerized 'billing system 
now, the findings we presented in this chapter could serve as a baseline 
Against which to compare the results of future sales analyses provMed by 
computerized sales records. In considering the cost— effectiveness of this 
option, WEEAP will of course weigh it against the likelihood of future 
appropriation levels for the program, the likelihood that the future will 
find the federal government still actively involved in sponsoring the devel- 
opment and dissemination of educational approaches, and the potential for 
enhancing the quality and rationality of future policy decisions by provid- 
ing policymakers with more reliable information on the results of WEEAP 
product dissemination.. These considerations notwithstanding, the financial 
investment required to computerize the rjales records of the WEEAP Publishing 
Center may be justified by the potential benefit to program management that 
would accrue from a system of continuous monitoring of sales performance. 
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Qualitative Information Needed to Illuminate Quantitative Analyses * 

4 If WEEAP Is Interested In why certain Items sell, and why certain pur- 
chasers buy, It will be necessary to obtain this Information from data not 
presently available* One Idea would be to pre-print a few pertinent ques- 

* 

tions on the catalog order form* Either closed or open response choices 
could be provided for and the data could be coded an4 entered on the com- 
puter at the time the order Is filled* Then, the Information needed to 
answer certain "why* questions would be readily available in an immediately 
usablfe form for future sales analyses* 
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Chapter IV 
IMPACT FROM GRANT PROJECTS 



^ : ^ Purpose 

We wanted to obtain readily available information on WEEAP project^and 
product impact. Specifically, we sought existing evidence of the effective- 
ness of WEEAP funded and developed products and strategies. To obtain these 
data, we spoke at length by telephone with nine previous WEEAP grantees. 
These telephone conversations enabled us to answer, for this sample of WEEAP 
grantees, the following questions: 

• What was the extent and type of grantee evaluations? 

3 

• Did grantee evaluations address WEEAP objectives? 

• What evidence did granteea have to show the impact of WEEAP 
products and projects? 

• What evidence did grantees have to show WEEAP 1 s contribution 
to educational equity and educational improvement? 

• What tefnds of "ripple effects" occurred? 

• What has helped and/or tendered the impact of WEEAP products 
and projects? 

• How could grantee evaluations improve? 

.« 

Procedure 



Sampling 

The sample consisted of nine completed grant projects. Each prpject 
was chosen because some evidence of impact could be provided and the 
products were thought by staff of the WEEAP Publishing Center to be among 
the highest quality and best selling materials to have been produced and 
disseminated as of the end of 1980. The sample represented a diverse set 
of projects, products, and user groups (see Table 21) ♦ However, the sample 
is neither random nor representative. While the findings we present cannot 
be generalized beyond these nine grantees, they served the following very 
useful purposes: 
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Table 21 



BRIEF DESCRIPTIONS OF SAMPLE PROJECTS/PRODUCTS IN THE STUDY SAMPLE (N-9) 



Project 



Brief Description 



1 

2 
3 

I * 

00 

5 

6 

7 
8 

9 



This project developed two sets of self-contained multi-ethnic curriculum materials for junior high school students on women In 
United States and World History. The units contain a student book and a teacher's guide. The teacher's guide Includes unit objec- 
tives and strategies for dealing with students. The materials were pilot tested and reactions were obtained from both students and 
teachers* 

This project developed a book ox activities designed to Increase students 1 awareness of career-related values, talents, and interests. 
The book also alms to help students make career and education decisions, particularly with regard to nontradltlonal options. The pro- 
duct was field-tested and reactions vere obtained from a variety of users. 

Thla project developed two reference books for women's centers. One of the books dealt with program planning; the other book dealt with 
budget negotiations. Both books offer complete "how-to" Information on these topics. A fleldtest was conducted to assess changes In 
the behavior and attitudes of persons using the materials. 

This project developed a guidebook for employment and career counselors who provide assistance to re-entry and underemployed women, wom- 
en seeking career changes, and women considering nontradltlonal career options. The guidebook outlines workshops, complete with ac- 
tivities and written handout materials., The book Is written with a wide audience In mind Including minority women. Reactions from 
workshop participants were obtained during pilot testing. 

This project developed a K-12 curriculum guidebook and audiotape cassettes on wouten In science and technical careers. The materials 
Include both historical Information as well as current science-related career Information. The materials were reviewed by experts 
In the field. 

This project developed a set of curriculum materials for middle-school students on the subject of career preparation and development. 
The materials Include a leader's guide, a gamebook of student activities, and a fllmstrlp and script about women in various careers. 
The materials were extensively pilot tested and field tested. 

This project developed a set of four booklets about the role of women In American history from 1607 to 1920. Each booklet contains 
discussion questions and activity suggestions. Reactions of students and teachers to the materials were obtained. 

Thla project develrped a film and accompanying factbook/dlscu5slon guide on the topic of nontradltlonal vocational education programs 
and careers for g'.rls. The materials are intended for parents as well as teachers and are appropriate for both classroom and com- 
munity use. A fleldtest was conducted to assess change in parents' attitudes toward their daughters' careers. 

This project developed a guide for workshop leaders who help re-entry women. The guide contains activities for job hunting and job 
finding. A fleldtest was conducted to assess the usefulness of the materials. 





ERIC 



• suggesting the nature of and reasons for' the very limited 
usefulness of "impact evaluation" attempted by WEEAP product 

• developers during their grant projects; 

• calling attention to "ripple effects" grantees mentioned 
without realizing that such outcomes are of great interest 
to WEEAP; a ad 

• providing a basis for WEEAP* s future evaluation planning. 



Data Collection and Analysis 

We conducted telephone interviews with 15 project personnel who were 
most knowledgeable about the nine projects and the products that had been 
developed. The interviews centered on grantee evaluations and product/ 
project impact, with particular emphasis on the questions listed at the 
beginning of this chapter. - 

Once the interviews were completed, we studied the detailed information 
for each project to note trends and patterns and to identify any interesting 
or unique exceptions. We then summarized the collected data in both narra- 
tive and tabular form. The next section presents our synthesis of this 
information, supplemented Sy illustrative examples of some of the results. 



Results 



We present the results in seven sections, one section for each of the 
questions listed at the outset of this chapter. Due to double counting, the 
items in the tables often add up to more than nine (the number of projects 
in the sample). ^In other words, some grantees were appropriately placed in 
more than one item or category. 

What Was the Extent and Type of Grantee Evaluations? 

All nine grantees conducted formative evaluations of their projects. 
Data were collected and used to assess and improve project functioning and 
to assist project management and product development. IMs formative pro- 
cess was "built into" most projects and was an on-going project monitoring 
activity^ In contrast, only six of the nine projects conducted a summative 
or "impact" evaluation to assess the effectiveness of the project in 
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' ' Table 22 

TYPES OF EVALUATION DESIGN FEATURES 



Design Feature Number of Projects (N ■ 9) 

Expert Review Only 1 

^cpert Review (In addition to other approaches) 8 

Post Assessment Only 5 

Pre and Post Assessment 4 

Use of Comparison or Control Group 2 

Follow-up Assessment 2 



achieving Intended outcomes. Of these six projects, all six assessed change 
In attitudes but only Tialf (3/6) of the projects assessed change In behav- 
iors. ^This may reflect the greater difficulty in behavioral assessment, 
which requires more training and more sophlsicated analysis, and for which 
fewer "packaged** measurements are available* 

None of the grantees were completely satisfied with the features of 
their evaluation design. They attributed limitations to events outside 
their control such as budgetary constraints, time constraints, and lack of 
expertise. 



Example: Even the project with the most sophisticated design 
had an objection. The project included expert review of 
materials, a pilot test, and a full fieldtest complete with 
large samples and a randomly assigned control group using both 
pre and post assessments. The project director revealed one 
major design flaw: the full "program" that had been developed 
had not been fully implemented during the fieldtest* According 
to the project director, this design flaw limited the potency 
of the fieldtest and potentially masked the full impact of the 
WEKAP product that had been developed. 

The level of effort devoted to evaluation activity varied widely among 
the sample projects. Rough estimates were given as follows: 
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Table 23 

ESTIMATED LEVEL OF EFFORT DEVOTED TO EVALUATION OF GRANT PROJECTS 



Estimated Level of Effort 
10Z or less 
30Z 
40* 

5 OK or more 



Number of Projects (N » 9.) 
3 

2 • • 
1 

3 • ' 



Six of the nine project budgets provided for an external evaluator or 
evaluation consultant. Three of the projects^ conducted the evaluation 
during a no-cost extension period. In two of these cases, the evaluation 
was seen as an "after- thought"; an evaluation was not originally included in 
the grant proposal. 



Did Grantee Evaluations Address WEEAP Objectives? 

The logic model of the intended WEEA program presented in Figure 1 of 
Chapter I identified seven outcomes expected to result from grantee proj- 
ects^ Table 24 identifies the number of grantee projects that actually 
focused their evaluation on these outcomes. As can be seen in tRis table, 
all grantees developed some model product or strategy. Most projects imple- 
mented their product or approach and attempted to assess whether those 
exposed to the product or approach benefited. None of the grantees reported 
that their evaluation focused on "improvement resulting from technical 
assistance." However, since this sample was selected in part for the excel- 
lence of its products, these grantees may not have required such assistance. 
Although about two-thirds to three-fourths of the projects focused on most 
outcomes of interest to WEEAP, our perception was that they seemed to do 
this because of their own interest, not because they knew what WEEAP wanted 
them to focus on in their evaluations and not because they knew about 
WEEAP* s overall program objectives. 

> 



-61- 

ERJC 73 



Table 24 

NUMBER OF GRANTEE EVALUATIONS THAT ADDRESS 
OUTCOMES WEEAP EXPECTS FROM GRANT PROJECTS /PRODUCTS 

Number of Evaluations 
Addressing the Outcome 
Expected Outcomes (N » 9) 

Target groups, staff, and relevant others % 
benefit from exposure to the model projects, 
materials, or approaches (Box 13)* 8 

Conducive environment for change and ability 

to generate change fostered (Box 14)* 6 

Capability of educational systems to work for 
and achieve educational equity enhanced 

(Box 15)* 6 



Educational leaders implement' the model 

projects, materials, or approaches (Box 16)* 

Diverse model products and change strategies 
produced (Box 17)* 



Evidence of effectiveness from grantees 1 tryout 
of model projects, materials, or approaches 
is available (Box 18)* 6 

Grantees 1 performance improved through tech- 
nical assistance (Box 19)* 0 



* Numbers in parentheses refer to the numbered "expected outcomes" for 
grant projects shown on the Detailed Logic Model of the Intended WEEA 
Program presented in Figure 1 of Chapter I. 

What. Evidence Did Grantees Have to Show the Impact of WEEAP Projects and 
Products? 

Each grantee reported some evidence of direct impact. Table 25 presents 
the general types of expected and accomplished outcomes. All grantee proj- 
ects expected to develop clear, usable, flexible, and attractive products. 
This is not surprising since developing model products, projects, and 
approaches is a grant requirement. Two grantees conducted a formal needs 
assessment in order to insure greater product usability, accuracy, clarity, 
and attractiveness. Every project claimed some evidence that the developed 
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Table 25 

EXPECTED AND ACCOMPLISHED OUTCOMES OF GRANTEE PROJECTS 



Outcomes 

* 

Develop products that are clear, usable, 
flexible and attractive 

* Produce attitudinal change reflecting posi- 
tive views of women, women T s roles, 
women's careers, women's issues 

Produce behavioral change reflecting 
increased skills and abilities 

Train others in the use of materials 

Increase program planning ability and 
increase funds for women's centers 

Interest boys in suppqjtffng* equity concerns 



Number of Projects (N ■ 9) ' 
Expected Accomplished 



3 
3 

1 
1 



2 
3 

1 
1 



materials were favorably received by the target audience. Three grantees 
noted that male teachers, students, counselors, and parents were more criti- 
cal of the products than were their female counterparts. In two of these 
cases, product revisions included efforts to incorporate tuore materials 
related to boys. 

As seen iji Table 23, one-half of the projects that expected attitudinal 
and behavioral change actually demonstrated such change. Examyles of spe- 
cific accomplished attitudinal changes included increased positive attitudes 
toward women in the work force and in non-traditional occupation? and 
decreased sex-typing of roles. Although attitudinal change was found among 
students, teachers, counselors, and parents, it should be noted that females 
changed more than males. Some examples of student attitudinal change follow: 

Example : One project found that girls exposed to the developed 
materials were significantly less sex-typed in their attitudes 
toward education and work than girls not exposed to the materi- 
als. Despite the finding that boys liked the materials as much 
as girls, those boys expo3ed to the materials were not signifi- 
cantly different in sex-typed attitudes than boys not exposed 
to the materials. 
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Example : Another project- assessed the effectiveness of their 
materials' in changing parent attitudes toward their daughters 1 
pursuit of nontr^ditional careers. More specifically, assess- 
ments were made birth before and a^ter the program and included 
an examination of course selection and career interests of 
daughters; awareness of the work-life potential of their 
daughters; and encouragement of daughters to explore nontradi- 
tional courses and careers. Parent attitudes in the three f 



Example : Another project found no overall significant differ- 
ence between groups who participated in U.S. history courses 
that included women in the curriculum and those in U.S. history 
courses that did not. However, student attitudes toward male 
and female natures, behaviors, and extraf aWlial roles were 
less stereotyped if their teachers had participated in the 
inservice training program and used t'le curriculum materials. 

Behavioral change was much less frequently explored. Of the three 
projects that assessed behavior, two projects demonstrated change. One 
project reported increased negotiations and communication skills; the other 
project reported increased job hunting and job-finding skills. The studies 
which assessed behavior did not use controlled ^research designs. The data 
were basically subjective or testimonial. 

All three projects that expected to provide training did so, and 
continued use of the products was noted in all three of these cases. The 
one project that expected women's centers to increase their ability to 
obtain funds found that, of the 12 women's centers they were able to con- 
tact, 3 had received substantial funding increases. 

Only one project assessed boys' interest in supporting equity. In that 
evaluation, females changed more positively than males; males did not get 
"worse," but in some cas^.s their attitudes remained unchanged. Also, boys 
responded less favorably than girls to the materials during pilot and field- 
tests. Grantees in our sample reported that one general problem to date 
with WEEAP products has been that they are not as amenable for use with boys 
as with giris. * ' 

Many grantees reported that their inability to demonstrate more dramatic 
change was the result of methodological or design problems in the evaluations 
and not of weaknesses in their products. Our conclusion from reviewing 
these evaluations was that there were many design flaws which could obscure 
subtle, and not so subtle, effects. ^ 



areas significantly increased. 
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Grantees used a variety of different indicators as measures of impact 
.in their evaluations* The number of indicators used in a single evaluation 
ranged from one to. five, with an average of two indicators per evaluation. 
Table^26 summarizes the general types of indicators. As Table 26 shows, 
almost all of the grantees devised their own indicators for their particular 
assessments. In mast cases, these instruments included rating scales and 
open-ended questions specifically related to the developed materials. Most 
of these indicators were self-report "paper and pencil" measures, although 
several grantees did conduct some on-site observation. 

The data sources were either the expert/target group reviewers or the 
pilot/fieldtest participants. None of the projects provided for systematic 
review of documents or existing records. H6wever, several projects cited 
^impact evidenc^v documented by personal correspondence. 

\ '. V 

Table 26 

INDICATORS USE3£ IN GRANTEE EVALUATIONS 

Number of Evaluations 
in Which the Indicator 
Indicators * ' Was Used (N - 9) 

Questionnaires or interviews developed specifi- 
cally for the purposes of the study 8 

Self-report "paper and pencil" methods 8 

On-site observations 5 

Unsolicited phone calls or correspondence 5 

Follow-up phone Later views with users ^ 2 

Rating of videotapes 1 

Standardized educational or behavioral science ; 
measure ^ 

Feedback from expert reviewers only 1 



ERLC 



What Evidence Did Grantees Have to Show WEEAP's. Contribution to Educational 
Equity and Educational Improvement ? 

Grantees identified two major areas of contribution to the overall 

quality of educational programs, policies, and practices: (1) increased 
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awareness of women's educational equity issues and (2) incorporation of 
grantees 1 product(s) into existing or new courses and programs. With regard 
to the first contribution, grantees claimed increased awareness among a 
variety of groups including students*, teachers, counselors, parents, and 
administrators. Some of this awareness was quite personalized, e.g., work- 
shop participants became more sensitive to their own personal struggles and 
evaluated their own attitudes and behaviors. 

Example : In one project that conducted teacher training, 
classroom observations were role-played and videotaped. Upon 
reviewing the videotapes, many of the teachers realized 
implicit or covert sexism in their pwn teaching styles. 

Example : One project found that middle school students were 
very interested in talking about themselves and their futures. 
The students found that the WEEAP developed program and materi- 
als gave them an opportunity to at least talk about sex equity 
and how it may affect them and their education or career in the 
future. 

In other cases, the increased awareness was more global and involved an 
introduction to the issues of sex equity. 

Example : One project involved university administrators in 
their workshops • The administrators were used as "helpers' 1 
rather than participants (the participants were directors of 
women's centers). As a result of their contact with the 
project, the administrators claimed that they, for the first 
time, became sensitized to the real seriousness of the equity 
issues. 

The second area of contribution, the incorporation of products into 
existing courses or programs, was reported by five of the nine projects. In 
two cases, a new course was developed on the basis of the WEEAP product. 



Example : A vocational-technical school in the midwest adopted 
the WEEAP materials and developed a course in which to use the 
materials* They then developed- a workshop for course members 
to explore further the application of the materials in real 
life situations. 
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Example : A western public school district adopted the WEEAP 
materials for United States history classes. Given a positive 
reception from both teachers and students, the same materials 
were then adopted for use in World History classes as well. 

* * 

What Kinds of Ripple Effects Occurred? 

This question addresses the longer term impacts of the WEEAP program. 
Grantees were asked to recall unintended activities, events, or M spin-off M 
experiences that occurred during- and/or after the WEEAP grant. Forty-eight 
different ripple effects were reported (an average of five per grantee). 
The general types of effects are shown in Table 27. 

\ Table 27 

TYPES OF REPORTED RIPPLE EFFECTS 

Ripple Effect Number of Projects (N » 9) 



Requests for Materials . * 9 

Use of materials with audiences not 

originally intended 6 

Media coverage/presentations 6 



New programs or new funds to continue or expand 
the original grant project 

Recognition of product excellence with presentation 
of an educational award 



Use of product as a "model" of non-sexist curriculum 2 

Improved student scores on state achievement exami- 
nations for questions dealing with women 1 

Expansion, of professional networks 1 

.Self-assessment by students of other aspects of 

their personal lives (values, beliefs, etc.) 1 



All grantees reported requests for their materials* In most cases, 
these requests were referred to the WEEAP Publishing Center. The grantees 
^ho kept records of requests found that they came from a wide variety of 
individuals and groups* 

Example : One project director asked the WEEAP Publishing Center 
. for sales records for her product for a one-month period. The 
grantee tabulated the list of buyers by rtype." Contrary to 
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her expectations, the products were requested by an exceedingly 
•wide range of user types* 

Of the six grantees reporting that their materials were being used with 

m 

audiences not originally intended, four grantees reported ^materials being 
used in more grade levels. Materials intended for school use were reportedly 
used' also in non-school settings such as churches, YWCAs, and prisons. 

Media coverage/presentations were reported by six grantees. This 
coverage reached both professional and popular audiences. 

Six grantees received funds to continue their equity activities. In 
one case, the new project was a national replication of the original WEEAP 
grant. In other cases, the new project was only tangentially related to the 
WEEAP project but remained in the sex. equity field. It appears that a # 
grantee's success in obtaining new resources for continuing equity activity 
is a potentially important ripple effect from having completed a project 
funded by WEEAP. 

Several of the reported ripple effects could not be categorized within 
the classifications used for Table 27, but are nevertheless noteworthy. 
Three examples follow. 

Example : A grantee wrote curriculum materials on women's his- 
tory for the WEEAP grant. After the grant, she was hired by a 
publishing company to review new textbooks for adequate cover- 
age of women. As a result, this grantee believes she can insure 
infusion of material on women into the curriculum materials. 

* * * 

Example ; A grantee reported a curriculum change which occurred 
immediately following the completion of the WEEAP grant; A 
high school home economics course was expanded to include 
males, changed its title to "Adult Living," and reorganized its 
focus to include a variety of independent living skills. 

* * * 

*** * 

Example Upon the completion of the WEEAP grant, a project 
director consulted with a large community prograjn. A handbook 
and manual were developed which incorporated parts of the WEEAP 
grant^ products .in an adapted form. The community organization 
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manual was disseminated to a large and varied audience and was 
received by persons who would, not have purchased the original 
materials from the WEEAP Publishing Center. 

» 

What Has Helped and/or Hindered the Impact of WEEAP Products and Pr jects? 

All grantees were able to identify factors that contributed to or 
inhibited product impact. Contributing factors are presented in Table 28. 
Expertise and commitment of staff were identified most frequently as con- 
tributing factors. Expertise took the traditional form of expert knowledge 
in a content area as well as less conventional forms of "expertise." 

Table 28 

FACTORS CONTRIBUTING TO PROJECT/ PRODUCT IMPACT 

Contributing Factors Number of Projects (N ■ 9) 

Expertise and commitment of staff 4 

Assistance from other organizations or groups 2 +/ 

Positive teacher attitude 2 

x\p pro priate, timing of events 1 

Inte«*ed product implementers involved at outset 

of project 1 



Example : One grantee reported that she felt the staff's sensi- 
tivity, as well as their ability and willingness to get teachers 
to be frank about their concerns were critical contributing 
factors to product and project impact. Once the teachers were 
"given permission" to voicl* their concerns, they willingly 
adapted and used the materials in their classrooms. 



Example : One project director said she was very open with 
workshop participants about her own personal struggles with sex 
equity issues. She felt the audience d could identify with her 
as a role model because her examples were grounded in personal 
experience. 

* 

Two grantees reported that assistance from other organizations and groups 
were critical contributing factors for impact. Helpful organizations 
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mentioned included CETA, YWCA, and community colleges and their extension 
services* Additional contributing factors included positive teacher atti- 
tude, the fact that the project was timed appropriately, and the involvement 
of the intended users of the product at the outset of the WEEAP grant. 

Grantees identified several factors which inhibited the impact of WEEAP 
pro jects/products. These inhibiting influences are summarized in Table 29. 



Table 29 

FACTORS INHIBITING PROJECT/ PRODUCT IMPACT 

Inhibiting Factor Number of Project (N » 9) 

Grantees' lack of control over the dissemination 

of their products 4 

Limited tims and /or funds for doing the grant 3 

Grantees' lack of face-to-face contact with 

users of their products 3 

Negative teacher attitudes 2 

Difficulty in changing attitudes 2 

Influence/non-support of local politics or local 

school district 2 

Inadequately planned use in educational setting 1 

Visual unattractiveness of product 1 

Difficulty in finding adequate outcome measures 1 



The four grantees who complained of their lack of control over the 
dissemination of their products expressed these concerns about the adequacy 
of product dissemination via the WEEAP Publishing Center: 

• Marketing has been insufficiently tailored to potental users 
for the particular type of product. (For example, the WEEAP 
catalog contains all products and is a major vehicle for 
disseminating printed information about available material.) 

• The major target for marketing WEEAP^ products has been 
educational institutions and personnel. More effort should 
go into broadening the base of potential users. 

• WEEAP products have time value and demand quick dissemina- 
tion so they will not be outdated before being widely 
distributed. More effort must be devoted to shortening the 
production schedule by the WEEAP Publishing Center. 

p <> 
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Three grantees were concerned about the lack of face-to-face contact 
with users of the products they developed. They saw such contact, particu- 
larly if it were in the form of training in which buyers get first-hand 
experience in using ^he materials, as a way to enhance sales and greatly 
increase the likelihood of product utilization. 

Three grantees acknowledged that limited time and/or funds seriously 
inhibited the potential for impact from WEEAP products and projects. Two 
grantees mentioned the detrimental impact on students' experience with WEEAP 
materials that is likely if teachers *te not receptive to WEEAP products. 

Two grantees reported the inf luei of local or in-house politics as an 

inhibiting factor. 

Example : One local School district required the WEEAP grantee 
to employ a school district evaluator. This evaluator knew very 
little about sex equity issues and, in fact, seemed unreceptive 
to evaluating the types of outcomes and impacts the grantee was 
hoping to find. The grantee said that the evaluator f s lack of 
support detracted from the quality of the evaluation. 



How Could Grantee Evaluations Improve? 

All nine grantees identified some way in which WEEAP could enhance the 
quality and soundness of grantees 1 evaluations. These suggestions are pre- 
sented in Table 30. The most frequently reported recommendation for improv- 
ing grantee evaluations was for WEEAP to provide more technical assistance 

Table 30 

RECOMMENDATIONS FOR IMPROVING GRANTEE EVALUATIONS 



Recommendations Number of Project (N * 9) 

Provide more technical assistance on research 

and evaluation 6 

Provide funds to collect follow-up <iata 1 

Set minimum standards of acceptability 

for evaluation design 1 

Provide means of basing evaluations on diverse 

populations in various parts of the country 1 

Encourage more frequent use of on-site 

observation data 1 
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in research and evaluation issues and problems associated with grant proj- 
ects* Many grantees felt the size of their grants was insufficient for sat- 
isfactorily remedying some of the evaluation issues and problems that arose. 

Example : One project director of a small grant went to the 
WEEAP technical assistance conference. She was disappointed 
because she thought evaluation information presented was lonly 
relevant to much larger projects. She wanted to learn about 
how to conduct a good evaluation with very limited resources. 

None of the grantees reported a clear understanding of WEEAP 's specific 
needs for evaluation information . Therefore > they could not respond to our 
request for suggestions on improving the pertinence of grantees' evaluations 
for WEEAP 1 s information needs. 



From telephone interviews with nine previous WEEAP grantees, we col- 
lected and examined information related to the impact of WEEAP projects and" 
products. Although all grantees conducted some form of evaluation activity, 
their evaluation research designs rarely provided for controlled studies, 
and over half of the projects failed to collect any baseline or pretest 
data* Most of the indicators of impact were limited to questionnaires 
designed specifically for the purposes of the study* All of the grantees 
were acutely aware of these limitations, which they agreed had limited their 
ability to demonstrate project and product impact. The need for technical 
assistance was their most frequent recommendation for improving grantee 
evaluations in the future. 

Outcomes of most interest to WEEAP were addressed in grantees' evalua- 
tions, but often only indirectly. Particularly in the area of attitudinal 
and behavioral change^ the impact grantees expected exceeded what was accom- 
plished. Grantees were able to provide some evidence of increased awareness 

I 4 

of women's educational equity issues and the incorporation of WEEAP products 



* For the last two years (fiscal years 1980 and 1$81), individualized 

technical assistance on evaluation and research design has been offered at 
WEEAP's conferences for its project directors. 



Summary 



into existing or new courses and programs. 
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variety of "ripple effects" grantees reported, e.g., requests for materials, 
use and adaptation of materials with a variety of ' audiences other than the 
target audience, media coverage/presentations, and the development of new 
programs or new funds to continue or expand women's educational equity 
activities. 



Implications for WEEAP's Future Evaluation Planning 

We conclude this chapter with our judgments of: 

(1) problems that will continue to undermine the quality and 
pertinence of grantees' evaluations unless WEEAP takes 
steps to remedy the underlying reasons for the problems; 
and 

(2' recommendations for improving the feasibility and useful- 
ness to WEEAP of grantees' evaluations in the future. 

Problems of Quality and Pertinence in Present Evaluations of WEEAP Grants 

The major problems we noted in this sample of grant project and product 
evaluations were as follows: 

• Grantees did not intentionally plan evaluations that would 
measure outcomes of most interest to WEEAP, and did not 
present findings in the context of WEEAP's overall objec- 
tives or information needs. 

• H Ripple effects," or unplanned outcomes, from doing the 
grants were pertinent to WEEAP's objectives and information 
needs, but in general were not included in grantees' written 
reports of their project and product evaluations. 

• Impact or effectiveness data were very limited and idiosyn- 
cratic Each WEEAP grantee addressed project-specific out- 
comes and used different indicators. Taken as a group, 
their findings, outcomes, and effects were often non-corrob^ 
orating anid, at worst, contradictory. 

I 

• Very few grantees conducted an assessment of their projects 
or products which would meet even minimum standards for 
sound evaluation design. Consequently, the results are 
difficult or impossible to interpret. 

Wh y did these problems occur? First, grantees were virtually unaware 
of WEEAP's overall objectives, the outcomes of grant projects that would be 



of most interest to federal program management, and how WEEAP would actually 
use such evaluation findings if grantees could provide them. 

Second, grantees seemed only vaguely aware of basic aspects of evalua- 
tion design (e.g., use of control groups and of pre and post assessment), 
and very few actually implemented these features because of either limited 
funds, expertise, or time. Some grantees were unsure of how to incorporate 
such design features into their own project and product evaluations. 

Third, grantees were often unsure about where to go for help. Evalua- 
tion assistance from WEEAP project monitors was said to be insufficient. 

Fourth, grantees fe?t so inundated with product development and grant 
administration responsibilities that evaluation was oftengiven lower 
priority. 

Recommendations for WEEAP's Future Evaluation Plannin g 

In our judgment, WEEAP can greatly improve the feasibility and utility 
of future evaluations of grant projects and products by doing the following: 

1. Tell grantees exactly what outcomes and information needs 
are of most current interest to program management. 

2. Specify in this communication how WEEAP intends to use this 
information, including examples of how similar data have 
been used to advantage in the past. 

3. When providing grantees with modest amounts of technical 
assistance in evaluating their projects and products, empha- 
size the use of economical and readily available indicators 
and capturing outcomes that may seem trivial to the speci- 
fic project /product but can be compelling when corroborated 
by similar effects from other grantees' evaluations (e.g., 
ripple effects like those teported by the sample of grantees 
in this study) . 

4. Confine the federal program investment in collecting new 
data to a limited number of grants whose project staff have 
adequate expertise for designing and implementing rigorous 
evaluations of their projects, products, and approaches. 
These grant projects should be adequately funded for a 
formal evaluation effort, and that activity should be 
regarded as an integral part of the project. Tha outcome 
measures employed should.be directly related to WEEAP r s 
obje ctives and information needs , in addition to project- 
ant, product-specific assessment. 

The financial outlay required for implementing the first two of the four 
recommendations is negligible, since communication between WEEAP and its 
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grantees can occur in negotiations of grant awards, letters to grantees, and 
regularly scheduled meetings for directors of WEEAP grants and contracts. 
The cost to WEEAP for implementing the third recommendation could be mini- 
mized. For example, a technical assistance contractor, funded at a modest 
level, could make liberal use of mail and telephone for helping grantees . 
with their evaluation problems* A directory of former WEEAP grantees with 
evaluation expertise could be used by the contractor and the current grantees 
as an additional source of low- or no-cost help. 




Chapter V 

FACTORS INFLUENCING THE UTILIZATION OF WEEAP PRODUCTS 
AT FIVE NATIONAL DEMONSTRATION SITES 

Purpose 

In this part of the rapid feedback evaluation, we sought to identify 
factors that influenced the utilization of WEEAP products and approaches in 
comprehensive educational programs and to describe their consequences. AIR 
conducted interviews with representatives of the f ive WEEAP national demon- 
stration sites, focusing the interviews on the following questions: 

Product Choice 

• What types of products were selected for use at the demon- 
stration sites? 

• Did the demonstration sites turn to other sources of equity 
materials? If so, why? 

Product Utilization 

• How were the products used? 

• Were the products easy to adapt and implement? 

• Did use of the products require special technical assistance? 

• What were the material and non-material" factors affecting 
product utilization? 

Product Quality 

• What was the technical and substantive quality of the 
\ products? 

Future Directions 

• Do thk demonstration sites plan to continue using WEEAP 
products? } \ 

I 

• What kinds of educational equity products should WEEAP 
develop in the future? j 

• Is WEEAP's current dissemination system satisfactory? 



• What type of technical assistance in product utilization dc 
the demonstration sites prefer? 4 

• Are the demonstration cites inter°sted in continuing or 
endorsing educational equity activity in the future? 

• What are the factors encouraging educational equity activity 
at the demonstration sites? 

• What is the outlook for future educational equity activity 
at the demonstration sites in the absence of federal funds? 

• Is the "national demonstration site" strategy desirable and 
appropriate? 



Procedure 

The sample consisted of WEEAP's five national demonstration sites. 
What is a national demonstration of educational equity? Five school dis- 
tricts under contract with the Women* s Educational Equity Act Program have 
been funded for the primary purpose of developing and implementing a compre- 
hensive equity program throughout the school district. Specifically, the 
sites do, the following: 

• Introduce materials that will contribute to achievement of 
an environment devoid of sex and ethnic bias. 

• Provide a setting where visitors from throughout the nation 
may observe equitable practices. 

• Train local personnel to infuse equity concepts into their 
daily activities. 

• Offer training to off-site individuals interested in estab- 
lishing equitable programs in their own locales. 

i 

i • Evaluate program effectiveness. j 

Utilization^ of WEEAP products is only one activity of a much latger', 

\ i - \ \ 

comprehensive equity program at the demonstration sites. Utilization of tjhe 

products, however, wa£ the focus of the interviews conducted at the five 

i 

sites* j 

The five sit^s represent a broad range of school systems from across 
the country, including urban and rural areas, and student populations ranging 
from barely 5,000 to over 100,000. The sites selected by WEEAP for the 
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national demonstration have shown an interest in sex equity through existing 
programs and support and have a successful track record for accepting and 
disseminating innovative educational programs and materials. The following 
thumbnail sketches. are intended to convey the highlights of the national 
demonstration th^t are unique to each of the five sites. 

Broward County, Florida ; .,, The National Sex Equity Demonstra- ■ 
tion Project functions in the Broward County School District. 
It is a cooperative effort of the district and the University 
of Miami. The public school system is the tenth largest in 
the nation with over 134,000 students and 12,000 full-time 
employees. Seventy-four percent of the students are white, 
with 23 percent black, and 3 percent Hispanic. The district 
has 97 elementary schools, 20 high schools, 27 middle 
schools, 9 exceptional student centers, 10 adult centers,. 12 
community schools, and 2 area vocational technical centers. 
^The project operates in 2 elementary schools, 1 middle 
'school, and 1 high school. The primary demonstration site 
is the Nova Research and Development Center — a staff and 
curriculum development center for the Broward County Schools. 
This project includes subject area workshops for staff of 
all grade levels. < A national advisory board of superinten- 
dents, equity advocates, and experts in school improvement 
provides leadership and support for the Broward County proj- 
ect activities. 



Lincoln County, Oregon : The Equity Project in Lincoln 
County, Oregon is a cooperative effort between the Lincoln 
County schools and the Northwest Regional Educational Lahr 
oratory in Portland. The Lincoln County School District 
serves about 5>000 students? 95 percent being white, 3 per- 
cent Siletz Indians, and 1 percent Pacific islanders. The 
county has 20 schools with 10 elementary schools, 4 middle 
schools, and 4 high schools, and two first through twelftl 
grade schools, ^here are 408 professional staff and 365 
classified employees. The project works intensively in 4 
schools — 2 elementary, 1 middle school, and I^high school. 
• In each of tlfe four demonstration schools, n community advi- 
sory board oversees tha prolject. A steering committee of 
teachers, aides, and adminilstratcrs plans equity activities. 
■Local parents learn about the project, at an Open' House, 
where they meet project staff and hear about classroom 
activities. Teachers can receive credit for taking project 
courses, which are designed to meet Oregon requirements that 
teachers be trained in anti-discrimination laws. 



Quincy, Massachusetts : Project Inter- Action is a joint 
project of the Quincy Public Schools and the NETWORK, a 

/ 
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non-profit educational service organization. The Quincy 
Public Schools serve about 12,000 students with 99 percent 
of them being white. The system includes 18 elementary 
schools, 5 junior high schools, 2 high schools, its own 
vocational-technical school, and a junior college. There 
are approximately 1,000 professional staff and 380 non- 
professional employees. The project operates in 3 elemen- 
tary schools, 1 junior high school, 1 high school, and 1 
vocational school. In these six impact schools, action 
teams of teachers, parents, and the principal plan a variety 
of equity activities. One special event organized by the 
vocational school action team was a seminar in which over 
300 students and parents visited the school and discussed 
non-traditional courses and job opportunities with voca- 
tional teachers and trades people. 



Reidsville, North Carolina : Project NEED (National Educa- 
tional Equity Demonstration) is run by the Reidsville City 
School District and tb<* University of Tennessee. There are 
4,730 students in the district with a white to black ratio 
of 56 percent to 44 percent. The district has 1 comprehen- 
sive high/ScKcol, 1 junior high school, 1 middle school, and 
5 elementary schools. The district has a staff of 264 pro- 
fessional /and 137 classified employees. The professional 
staff is 75 percent white and 25 percent black, and the 
classified staff is 60 percent white and 40 percent black. 
The project operates in two elementary schools, one junior 
high school, and one high school. In this demonstration 
site, all faculty members participate in several awareness 
sessions, and core teachers commit themselves to a semester 
of intensive L raining and to using equity materials in their 
classrooms. * Their classes are videotaped at the beginning 
and end of the school year and the tapes are analyzed by 
teachers and evaluators. 



Tucson, Arizona : FOCUS is a cooperative project of the 
Tucson Unified School District #1, the Fima County Career 
Guidance Project, and the American Institutes for Research 
of Palo Alto, California. The -school district is the 
largest/fin the state of Arizona and is/ organized into four 
regional administrative divisions plus-the central office. 
The District serves 57,000 students. There are 100 schools 
including^68 elementary schools, 16 junior highs, -and 9 high 
schools. ' Tucson/ is alco the home of the University of Ari- 
zona and' Pima Community .College. There is a diversity of 
cultural groups in the Tucson area wivh approximately .80 
percent of the population Anglo and the reipaindfer including 
Mexican- Anericans, Native Americans, and A6ian- Americans. 
The FOCUS project operates in 8 schools — 4 elementary and 4 
secondary. An important aspect of this # project has been the 
incorporation of bilingual materials and Mexican-American 
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role models. Teachers here can apply for FOCUS investments * 
to purchase equity materials, to develop learning activi- 
ties, or to conduct equity-related field trips. The Career 
Guidance staff conducts workshops and encourages teachers in 
Tucson to share successful approaches to creating more 
equitable classrooms. 

AIR conducted approximately two-hour interviews with representatives of 
these five site<? , Four of the interviews were conducted on site and one by 
telephone. For the most part, interviews were conducted with the project 
director at the contracting agency an<l the school-based coordinator at the 
demonstration site unless the project director suggested otherwise. e 
five demonstration sites and the position of each of the interviewees are 
listed in the Appendix. \ 

The interviews did not follow a standard set of questions. Rather, 
they were semi-structured conversations that focused on four general topics: 

• Product Choice 

• Product Utilization % 

• Product Quality 

• Future Directions 
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» Results 

The results of the interviews are summarized here according to the four 
general topic areas. Illustrative examples are also included for each of 
the results. 

Product Choice 



What types of products were selected for use at the demonstration a 
sites ? At all five sites, products were selected for ease of use in the 
classroom. These products had short activities and lesson plans, did not 
require a lot of teacher time to adapt! and were easy to pull apart and 
duplicate. At most sites, in-service materials were also selected for 
activities that could be adapted for classroom use wijth a minimum of effort. 
Although most sites ordered all types of WEEAP materials (resource materi- 
als, curriculum materials, staff training materials, audiovisual materials), 
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the most sought after and most used were the curriculum materials with the 
characteristics described above. 



"We ordered all the WEEAP materials. There are few enough so* 
reviewing them all was not such a job. Our Materials Review 
Committee, composed of teachers representing all subject and 
grade levels, was looking primarily for materials they could 
use in the classroom with minimal adaptation. We really need 
classroom materials with short activities." 



Did the demonstration sites turn to other sources of equity materials? 
If so, why ? When gaps in WEEAP products existed, demonstration site staff 
turned to a variety of other sources of equity materials* A major purpose 
of the demonstration sites is to use WEEAP products. Only in the event of 
gaps in these products did the demonstration sites turn to other sources of 
materials. These included state-developed materials, materials from com- 
mercial publishers, materials developed by various equity projects, and 
teacher-developed materials at the site. 



"We intended to use only WEEAP products, but there were gaps*-' 
mainly a lack of elementary-level curriculum materials. This 
was frustrating because we found the greatest, excitement 
about and the mort interest among school staff at this level 
in equity activities. We also needed a wider variety of 
materials than WEEAP has published so far.'V 



Product Utilization 

How were the products used? WEEAP proiucts were consistently used in 
part rather than in their entirety. At all five sites, the products were 
used primarily for the classroom activities that could be selected and 
adapted from them. Very rarely was a product used intact. Inservice 
training materials also tended to b : e used in part rather than in their 
entirety. 
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"We have used parts of the products for staff development and 
parent demonstrations. But we primarily use. the materials 
for the activities in them which we can adapt for the class- 
room. Actually, the teachers are the ones who decide what 
they'll do with the materials, which is usually modifying 
them for use with their students." 



Were the products easy to adapt and implement? Ease of adaptation and 
implementation depends on the characteristics of the particular product. 
In the process -of reviewing WEEAP products for their possible use in the 
national demonstration, staff at all five sites anticipated that some prod- 
ucts would be easy to adapt and implement while others would be difficult. 
Thus, they tended to select products that could be easily adapted, and 
therefore, as expected, national demonstration site staff reported that 
these 1WEEAP products were easy to implement. 



"Some of the WEEAP materials are not easily adaptable, and we 
avoided those. We chiefly sought materials that could be 
modified and used easily. The ones that were chosen were, in 
fact, easy to adapt and implement." 



Did use of the products require special technical assistance ? At -11 
five sites, use of WEEAP materials did not require special technical assis- 
tance. However, most expressed that "it would be nice to have." Staff at 
one site expressed the greatest need for assistance in developing curriculum 
writing expertise, particularly in developing good lesson plans and adapting 
materials. Staff at another site found the materials they selected to £e 
easy to use, but some assistance would have helped them speed up the selec- 
tion process. 



"Teachers could manage pretty much on their own in adapting 
materials. It would have been nice, but not necessary, to 
have 'how-to-use ' discussions of materials." 



What were the material- and non-raaterial factors affecting product 
utilization? The factors inherent in the materials themselves that affected 
product utilization consistently included interest, visual appeal, organiza- 
tion, and ease of use in the classroom. . At all five sites, the products 
that were used most frequently were well-organized, attra6tive, interesting, 
and easily adapted for classroom use. Staff at one site emphasized that the 
ability to pull out one specific activity was^^rixlcal.. 



"The products we used most frequently were easily' adaptable, 
attractive, usable, had an index or other organizing system, 
and could easily supplement an existing lesson plan. 



The non-material factors that affected product utilization consistently 
included in-service training in the use of the materials by demonstration 
site staff. most sites, the materials 'most likely to be used by teachers 
were the on^s for which they- had received in-service training. Other fac- 
tors affecting product use included recommendations from other teachers about 
a particular 'product , receptivity of teachers to the whole idea of equity, 
changing the term "sex equity" to "educational equity," using a "soft-sell 
approach" with teachers i i getting them to use materials, tes^ehei=-ijivolve- 
ment >in adaptation of materials, and supporS^ of de monstration site st 





"Teacher in-service in using a product has really made a 
difference! The sharing and passing on of useful materials 
from one teacher to another has also been important." 



Product Quality 

What vas the technical and substantive quality of the products ? Tech- - 
ni_ _ quality of WEEAP products was rated average but improving* 'All five 
sites would prefer to see more professional looking products, although they 
emphasized that the newer WEEAP products are an improvement over the older 
ones. All sites expressed a desire foi^more interesting visuals and gtaph- 
ics, mpre ufee of color, more attractive) t^pe, better layout. Most sites 
said that the biggest problem was th'e^poor quality of the audiovi ual 
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materials. The films trips were : not of a professional qyality in earlier 
products, although they are improving now. The videotapes cannot bS used 
with the equipment in most schools, and the 'content of the videotapes very 
of ten .does not warrant use of this particular medium. 
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*"We would realty like more 'professional 1 . Ipcking products. ■ 
We'd like to see more use of color and more attractive type. 
With a few exceptifcns, £he print products are getting better. 
WEEAF's early, AV materials were really bad. The filmstrips/ 
-cassettes, are getting better, but the video.tapes have not 
been useful and couldn't be used with existing equipment in 
many school districts." * 



Substantive quality of the WEEAP products was rated above average. 
When asted to comment on any drawbacks they had encountered, demonstration 
site staf-f noted that there is a lot of repetition of the same ideas in 
various materials; sonfe materials., are narrow in scope; elementary materials 
are lacking; most materials deal with women and need to address men also; 
some materials are too general and are not geared to specific subject areas; 
.some materials have inaccuracies and judgmental statements; some materials 
are too technical end theoretical; some are too geographically specific. 
On the positive side, most sites described the materials as, being well- 
organized and logical, with good research support and documentation. 



"The quality of %he content is excellent, but there are a 
lot of gaps. Sometimes the content is too technical and . 
theory-oriented or too specific geographically." 



/ 



Future Directions 

Do the demonstration sites plan to continue using WEEAP products ? All 
five sites stated they will definitely continue td use WEEAP products, but 
they will order more selectively to" meet subject and grade level needs. By, 
now the five sites have identified what they consider to be the best products 
and uill concentrate on using these. If the national demonstrations are no - 
longer funded, the sites feel that the good WEJ2A* products will continue to 
be used* 
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"We will continue to use the WEEAI* products that are already 
being used. If the' project dies, use of the good products 
will survive." * 



What kinds of educational equity product's should WEEAP devel6p in the 
future ? Although all sites have a variety of suggestions for future devel- 
opment of WEEAP products, all cited the need for more classroom materials at 
the elementary level and for more curriculum materials in general with 
short, easily adapted and easily infused activities. 



"There are giant gaps. We need more manipulative and visual 
things (games, posters) at the elementary level.' In fact, we 
* . 'neeti* mote materials in general at this level. We'd also like 
materials to be organized by content or subject area ." ; 

Table 31 presents the preferences of the demonstration* sites for WEEAP's 
future product develJ^S&nt • 
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Table 31 ' 

PFEFERENCES 0? DEMONSTRATION SITES FOR FUTURE PRODUCT DEVELOPMENT* 



■ Typ*e of Product Needed Frequency 

Materials Kith short , .easily adap'ted/infused 

classroom activities • 5 

^Materials with the appeal of ' commercially 
published products 3 

.* • "\ * 

Materials that are subject-oriented or are 

easily linked with subjects, topics, themes* * 3 
Materials with M how-to-use" information 

in them v 2 

Materials that identify .non-WEEAP materials 

that could be used with the WEEAP products % 1 

. Target Groups far Which More Materials are Needed 

Elementary grade students * t 5 m 

Males (Mo4t available WEEAP products were perceived 

as being female-oriented, limiting their interest for - 

school-age males.) 4 

Junior high school etudents 1 

Subject Areas for- Which More Materials Are Needed 

Math and/or Science * 3»* 

Vocational Education (beyond career education) , 1 

Other Needs 

More products from which to choose (quantity) " 1 



* Numbers refer to the frequency with which .the suggestion was mentioned at 
the five demonstration sites.*. 
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Ps WEEAP f s current dissemination system satisfactory .? Most sites are 
satisfied with the present centralized system of review and dissemination oi 
products through the WEEAP Publishing Center. However, it was hoped that 
the process of getting the materials published and disseminated could be 



9 

shortened. 



"We prefer a central system of Compiling and reviewing prod- 
ucts and believe^ it i? necessary, but the present system 
could be improved. The WEEAP product catalog isr a good dis- 
semination device, but the Publishing Center should get it 
out to a wider audience and should also be a vendor at all . 
the various' conferences. M 



s 



What type of technical assistance in product utilization do the demon- 
st ration sites prefer? Although most ^ites found they could manage using 
WEEAP products on their own, they said it would be useful to have* product 
developers and product users provide assistance both on site and at regional 
arid national conferences. Two sites also suggested tha't more h6w-to-use 
information be provided In the products, themselves and in the catalog of 
WEEAP products. * . r . 



— • • 

"We like the regional/national workshops sponsored by devel- 
opers. It teally helps to have the developer explain the 
product." s ' « * 

: — * : • . 

* „ * 

Ire the demonstration sites interested in continuing or endorsing edu- 
cational equity activity in the future? All sites aref strongly interested t 
in continuing and endorsdng educational 'equity activities. 

\ > 

/"We will definitely pursue equity. T^e principals and 
' teachers are very interested; ,in fact, more or them are 
interested than we can use in the project. We needed five 
teachers at a school this year, and eleven* signed upl M / 



What are the factors encouraging educational equity activity at the 
demonstration sites? Sites listed a variety of factors operating to tucour- 
age educational equity activities at the demonstration sites: 

* strong support from the State Department of Education 

• sophistication of school districts in terms of equity issues 

9 



• support of school teachers and administrators 
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• district-level support 

* • state laws and regulations supporting equity 
m availability of good equity materials 

• availability of role' modfels 

• 'community interest 



"Some encouraging factors are the positive experiences and 
success stories we hear from teachers .and principals* There . 
is also the pride^of being, a demonstration site. There seenjs . 
to be interest ia the community, and business JLs also 
involved. We've also seen some attitudinal chang'e on the 
part of school administrators." . 



What* Is the outlook for future ^educational equity activity at the v 
demonstration sites in the absence of- federal funds? Should WEEAP funding 
for the national -demonstrations end, staff at the five sites are not opti- 
mistic about their ability to sustain the current level of effort, not only 
in the utilization of WEEAP products but iri all demonstration site activi- 
t&es. Staff do anticipate the continuation of educational equity activities * 
through the teachers they have *traine<T. 



"The teachers whet are using the materials will probably con- 
tinue to promote educational equity activities in the class- 
room. If th% teacher turnover rate is high, however, project 
impact may be nil." t x 



Is the "national demonstration site" 'strategy, desirable and appropriate ? 
Most eifces think that the "National demonstration site" strategy is a good 
one and wish that it could be expanded in terms of time and resources. Spe- 
cifically, staff at most of the, sites- would have liked to have more planning 
time than the one year they were provided before implementing the demonstra- 
tion* . They also recommended that WEEAP plan a strong dissemination effort 

to promote the spread of similarly comprehensive educational equity program^ 

m t % 

beyond these five sites. * ' 

- -ioo • . 



* 1 

H We f re very supportive of this strategy I The idea of the 
*projjfct being in the schools is an important one. It's a 
very logical approach to -change-- work -from the top down as 
well as from the .bottom up." 



Summary ... 

Results of the interviews with the staffs of the five national demon A 

* < ■ * 

strafcion sites indicate that WE RAP products* were consistently chosen for 
ease of use \xl the classroom. Curriculum materials with short,' easily 
adapted and easily infused activities were the most* sought after, and the 
most used. Unirortunately , however, there are not^enough of these types of • 

^^aterials in the WEEAP product line at present., and 'the demonstration sites 
had to turn to other sources of equity materials to fill the gaps. 

The demonstration sites used the materials in part rather than in their 
entirety by selecting and adapting classroom activities from them. WEEAP' 

• products seem Vo vary in their ease off adaptation and implementation, but ( 
site staffs tended .to pick products, that would be easy for their teachers t,o 
adapt and implement.^ Use the materials -did not require special technical 

, assistance, but most s/Ltes said that assistance from the product developers- 
and other product users would.be nice to have. Thte most important factors 
that seemed to promote selection or use of the materials included in-service 
training of teachers in use p*f a particular producti and specific features 
of the materials themselves: interest, visual appeal, organization, and 
ease of use in the classroom. 

All sites felt the substantive quality of- the materials was better than 
the technical ^quality, but that. the materials werej^proving over early 
products in both re'garcls. All expressed special concern about the quality . 
of" the audiovisXial materials, and their perception^ that the videotapes are 
•not Compatible with 'existing equipment J.n many school districts. 

All five sites are strongly, committed 1 to educational equity activities 
and plan to continue using WEEAP products, particularly the^good ones tl>at 
• they have identified. They had a variety of suggestions for future develop- 
ment of WEEAP .products "but in particular cited a pressing need for more 
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cla.ssro<jp materials at the elementary "level and for more curriculum materi- 
als in general with short easily adapted and easily infused activities. 
Most sites were generally satisf ied with the current centralized WEEAP dis-j 
semination system, although they pres.sed for as speedier process in getting 1 
the materials published and disseminated. - 

Looking to the future 'and the possibility of no further sVpEAP funding, 
the sites anticipate the continuation of -educational equity activities 
.through the, teachers they have trained. Provided that resources are avail- 
able, -however, most site staffs would like to see the "national demonstra- 
tion site" strategy expanded* 



I mplications for WEEAP' s Future Evaluation Planning 

While Evaluation activities at each of the national demonstration sites 
differ in some respects, the five contractors are evaluating som^ commbn 
program components. Furthermore, 1 to the extent possible, they wiil .collect 
some common data elements. Qae of the five contractors is coordinating 
these evaluation activities with the intention of "providing for a "common . 
core of data" on various. facets of the demonstration enterprise. 

It is not yet clear to what extent the data from evaluations* being con^ 
ducted by the five sites can be integrated . When WEEAP has reviewed the 
•coordinated evaluation report, due next year, program management may judge % , 

that further analysis and integration of selected elements in the common^ 

» • *" » 

core of data will- be worth doing. 
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Chapter VI • 
SEA PERCEPTIONS , OF WEEAP IMPACT 

i 

% 

c Purpos e 

^ SEA experiences as ^EEAP. grant . recipients or as users of WEEAP products 
were ^investigated inthis part of the rapid feedback evaluation* Our purpose 
was to determine whether SSAs that have received WEEAP grants or* used WEEAP 
praducts found that these activities contributed to improved educational 
programs, policies; or practices. * w ^ 

The three major areas of" Interest were'as follows; 
.Impact from WEEAP's -Grants .to SEAs ' 

• • What were the intended outcomes from the WEEAP 'grant? , ' 

• We're these outcomes* achieved? 1 ' *■ 

* - 

• What post-grant activity' or events have occurred as a direct. 
1 ( result of the. WEEAf* grant- pro 3ect? • ■ 

■ • What evidence do grantees have to show the impact of the 
WEEAP grant" project? 

SEA Perceptions of WEEAP, Products' £nd Their Impact ^ • 

• What Yypes of products were purchased? * 

• How were the materials, used? 

.« * «. • 

. •» What were the > technical and substantive qualities of the 
•materials? 

• Were the materials easy to implement or to adapt? 

• What were the outcomes from using the materials? 

• What indicators were accepted as evidence of these outcomes?. „ 

Future Directions df Educational Equity Activity 

i * 

• Are there, plans for continued use of WEEAP products and 
approaches? . , 

• If federal funding for equity is included in block grants at 
some future time, what is the likelihood that the' state will 
allocate the present level" of funding for educational equity? 



If equity* material were no longer available through WEE AP ? 
would the SEA developiand publish similar'materials? ^ 

; • > . ' i . 

What factors encourage or* discourage educational equity 
activity of the SEA? t 

What could WEJEiAP do to- assist SEAs more with their educa- 
tional equity efforts? 



^Procedure 



Sample * * 

Nine SEAs were selected, based on indicators' of their past active 
involvement in educational equity. The indicators used as selection cri- , 
teria were as follows: • , 

• the SEA had received a WEEAP grant (category M a"); 

• the SEA had received Civil Rights Act Title IV, funding in 

1980-81 for sex equity 'activity and also fcas a WEEAP grant 

recipient' (category "b"); or * 
, . "it 

*• . the ; SEA'was a frequent purchaser of WEEAP. products according 
t£ WEEAP Publishing Center records and also was iEunded by 
Title IV in 'a 1980-81 for sex equity activity (category V*J. , 

Table 32 shows the number of states selected for each category by geographic 

region. m 

i * 

• . ' _ i> 

Table 32 

1 • • 

SAMPLE OF NINE STATES BY SELECTION CATEGORY AND REGION 



Selection Category , 

Region • ' (a) ' (b) (c) Totals' 

Northeast 1 .1 2 3 

North' Central 1 1 , 1 '3 

Southeast 1 .1 

West . J. 1 Lj± 

TOTALS ••'3.' 3' t 3 9 

General izability of results . The ggneralizability of findings, from 
surveying these nine SEAs is limited. As indicated, the sample was selected 



•because the SEA .had been a WEEAP grantee, or had been funded by Title IV for 
equity-related activity, or had been a relatively frequent purchaser of WEEAP 
materials. - Only 11 SEAs' Were eligible for selection, given these criteria, 
'and 9 of these were chosen for the survey. Thus, the findings to be pre- 
sented can be generalized only to the 11 SEAs* who were eligible for sample 

r • ^ 

selection, or about one-fifth of the total ntimbjer of SEAs in the country. 



Method # • * , 

Individuals to be interviewed were identified in the following manner. 
For the six "grantee states" (categories a and b), the former or current 
WEEAP grant project director was interviewed whenever that' individual could 
be located- In two" stafce.s, this- person Was not available '.and a x former proj- 
ect staff member was substituted. For the three "purchaser s±ates" (cate- 
gory c), the SEA office to be contacted was identified, from WEEAP Publishing 
Center records of purchases by state agencies. . * • 

The '"knowledgeable persons" to be interviewed were- identified by pool- 
ing information from several sources, as follows; . 

• suggestions from, the WE'EAP grank'e'Q or product purchaser; 

» ' , 

• leads 'prQjteLd'ed by* the Director of the Resource Center for , * 
Sex Equ^tySat the Council of Chief State School Officers in 

« ' Washington, dlC; \ • 

.leads provided in. a Janviafcy- 1981 j/emo from'^he Division of 
State Educational Assistance, Qffice of • the Assistant Secre- . \ 
•tary for Elemenpary and* §econdaVy' Education*, .U.S; Department 
of .Education, *A Look at?' Title IX^ Through Program Reviews of 
Part tf, Title IV ESEAJ" and 

• lists of, directors of Title IV (Civi^l Rights Act of M.964) , ' 
Desegregation Assistance Centers' arif Training Institutes, for . 
Sex EqGlty (Fiscal Yeax 1979). . 

To. Summarize, 18 persons employed^by*. % SEAs-were interviewed: two per- 
sons from each of -seven SEAs; three persons (dne purchaser arid ^Wo "knowl- 

"V * % " 

edgeable persons") from one SEA, and one. person fifom'an SEA whete the 

purchaser and tile "knowledgeable person" were the isame individual.^ See 
Table 33 for the positions of those interviewed. These were interviewees 1 
current positions, except in the case of the directors of completed WEEAP * 
grant projects. „ 



• •'. Table 33 * 9 . 

* . y 

. ^ POSITIONS* OF INTERVIEWEES 

(N «■ 18; duplicate count because of multiple positions held) 

" <• > ... \* 

* . Position* ' \. Number ■ 

State or Regional Leve l 0 f 

SEA sex equity" assistant^ coordinator y ■ ^ ^ 

' v SEA vo.catiorfal education equity ^coordinator \ \ . 2 

SEA intercultural relations/integration specialist *. 2 ■ 

SEA equal educatioa and l^egal specialist '* .1 

SEA occupational education/civil fights compliance director 1 

• SEA vocational development -specialist 1 
Sex Desegregation Assistance Center director " 1 



University 

^ " Eqdity center director' 
WEEAP Grantees 



Directors of completed WgEAP" grant- projects . 4 

Project associates of completed WEEAP grant projects * ^2 
Direjitor^ofc a current WEEAP 'grant pvofect ^ 4 i 

* These- were the current positions of interviewees, except as ndted. _ 

r" — ' r 

Telephone interviews lasting approximately one-t?alf hour w.ere conducted 
with the IS SEA employees* Interviews, were unstructured and focused 6n dif- 
ferent areas 'of inquiry ^depending -upon the category for which the SEA was 
selected and* the position of the "interviewee. Thus, interviews -With' the six 
WEEAP grantees focused on the questions ^related to WEEAP grant activity and 
impact and on future directions.. Interviews with 'the three WEEAP product 
purchasers focused on the questions related 'to impressions of WEEAP products 
and their impafct" and on fpture directions. And, in eight states, additional 
fnterrtews were conducted jjith nine person's .knowledgeable about the impact 
of WEEAP on the states ^educational equity activities in general, and these 
interviews- focused only on Ae questions related to future directions. 



Results 



■ Results of the survey of SEA perceptions of WEEAP's 'contribution .to 
improvement of educational programs, policies, and practices will be summar- 
ized in three" sections: 

• Impact from WEEAP 1 s Grants to SEAs 

• SEA Reactions to^ WEEAP Products 

• Future Directions 

t — 

In parentheses after each Sf these three headings we indicate the number of 
.SEAs on which the results are based. 




/ 



Impact from WKEAP's- Grants to SEAs QM? SEAs) 

What were the intended outcomes from the WEEAP grant proj<fct ? ^ Table 
34 lists. the outcomes the SEAs expected from their grant projects. In par- 
entheses after each outcome is the number of the box bn the Detailed Logic 
Model of the Intended WEEA Program to which ^this outcome relates. ^ (The logic" 
model A presented in Figure 1, Chapter I.) The only outcome WEEAP intends 
to result from its grants which was not mentioned by these SEA grantees was 
"Grantee's performance is improved through technical.assistancs" (Box 19). 
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Table 34 

OUTCOMES INTENDED 'FROM THE SEAs 1 WEEAP GRANT PROJECTS 



Outcomes Intended* 



SEA Grant Pro jects 
A B C D' E F 



TOTALS 



To develop or test SEA strategy 
to assist LEAs with sex equity 
(Boxes 13, 15 , 17) 



To improve educational equity at 
the administrative level th roughs 
out the state (Boxes 13, 14) 



to develop materials to build aware- 
ness and knowledge of 'sex stereotyp- 
ing at LfeA level (Boxes 13, 15, 17) 



To assist LEAs in starting 
educational equity projects 
(Boxes 13, 16) x 



To evaluate. effectiveness of 
materials .and approaches 
(Box 18) 



TOTALS 



14 



* Box. numbers in parentheses after each outcome listed in this table refer 
to the numbered outcomes on the Detailed Logic Model of the^ Intended WEEA 
Program wh5ch was presented in Figure 1 of Chapter I. 



jWere these outcomes achieved? Of the six SEAs who had WEEAP grants, 
five (had completed their projects and one was entering the final year of a 

hre|-year grant. Interviewees representing the five completed projects all 
said that their major goals* had been accomplished and that the benefits they 
could attribute directly to the project surpassed their original expecta- 
tions. Table 35 lists both positive and negative unforeseen additional out- 

comes SEAs noted as a result of their WEEAP grant projects. This table does 

y 

not break down these responses by grant project, but five of the six grantees 
interviewed cited 'generally increased awareness and understanding of women's 
equity issues among educators and students as an important result of the 
WEEAP grant project. 
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Table 35 

U FORESEEN ADDITIONAL OUTCOMES FROM SPAs' WEEAP GRANT PROJECTS- (N»6* SEAs) 

Unforeseen Positive Outcomes ' » Frequency 

Staff benefitted professionally from project participation # *2 

Value of SEA collaboration with other state agencies 

demonstrated 2 

Support for educational equity obtained from other key 

state agencies % 2 

Statewide network developed of individuals interested 
•» in Title IX implement a tioa and in sex equity 2 

Additional documents produced based on project activities 2 

Community awareness of women 1 s issues enhaiacad 2 

Community trust in goals of Title IX enhanced 1 

Student awareness of. women's issues enhanced ✓ 1 ^ 

4 

Private foundation funding obtained for equify 

activity in the state 1 

Hiring -of student workers stimulated ^ 1 

Unforeseen Negative Out cones * 

Males reacted negatively to equity material the SEA # - ^ 

grantee developed r ^ 

Users of the SEA grantee's material were mainly 

those already interested in women's issues (SEAs 1 

intention had been to increase such interest in 

users who were not necessarily "pro-equity .'*) 1 
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What post-grant activities have occurred as a direct result of the 
WEEAP grant? Of the five SEA grantees who had completed their projects, four 
reported significant post-grant activities that included continued widespread 
use of the developed materials in the state (4/5) and planning or conducting 
direct follow-up studies or activities (2/5). Only one of the SEAs said 
' that local interest in the materials they produced with their WEEA grant 
faded when the grant ended. Table 36 lists the post-grant activities that 
occurred after the SEA's WEEAP grant ended. 
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Table 36. 

POST-WEEAP GRANT ACTIVITY (N-5 SEAs) 



* . .Activity 



Continued use of SEA -developed WEEAP materials 
■ statewide/nationwide 

SEA staff regularly trained using project- 
developed materials ( . 

Training workshops conducted* by former project staff 

Follow-up activity funded and Operating 

Proposal written to continue g^ant activity 

Project participants assumed leadership positions 



Frequency 
4 



1 
1 
1 
1 
1 



$ What evidence do grantees have to show the impact of the WEEAP grant 
project? The interviewee in one SEA was unable- to provide information 'on 
the grant project f s evaluation, so responses summarized here are limited td 
five of the* six SEA grantees in our sample. 

These five grantees had gathered or planned to gather some evidence of 
impacwor of' the* effectiveness of their projects, and two of the grantees t 
were able to, measure the overall impact of their projects- or to attribute 
outcomes to specific project materials or strategies with some degree of 
confidence** Table 37 shows the kinds of evidence of impact that projects 
have or planned to have available and the number of projects that "have 
gathered or planned to-gather that . type of evidence. - 



110 



-100- 



' ' Table 37 

EVIDENCE OF IMPACT FROM SE As' WEEAP GRANTS (N-6 SEAs) 
Type of Evidence . t A . Frequency Cj.ted 

Datar from pre-post attitude survey > ' A 

(Compiled and analyzed) • m (3) 

(Uncompi\v>d) • - (1) 

Data from unstructured interviews with participants 2 
Personal 'impressions; notes, testimonies, anecdotes * A - X Z 

Data from pre-post assessment of ^changed practices 1 

► * 

Data from pre-post knowledge test * 1 

j 

• > 

Data from questionnaires on effectiveness of workshop x 1 

•Data from questionnaires on effectiveness of materials 1 

Comparative statistics on changes in training & • 

hiring practices ■ 1 

Statistics on numbers of persons'who participated in 

project activities _ ^ 1 

Continuation of program after end of graqt project ' 1 . 

The tTJo grantees who were convinced that their evaluation d/ita provided 

sound evidence of theimpact of their projects 1 activities used a pre-post" 

-evaluation design and/or comparison group design and assessment instruments 

they deemed to be valid and reliable. The evaluation strategies* of those 

two grantees are summarized 'below: ^ 

1. , Project objectives were clearly stated and an evaluation 
was planned to* address" each objective. Data w*re collected 
from participants statewide using (IX an attitude scale 
(pre-post), (2) a knowledge, test (pre-post), (3) workshop 
evaluation questionnaires, (4) materials evaluation ques- . 
tionnaires, and (5) structured interview questionnaires on 
overall project effectiveness, including information on 
course enrollment chang/es and follow-up of participants . 
Results showed* that attitude and knowledge had changed^in 
the desired direction in most groups, and that the changes 
were statistically significant. Positive response to work-< 
shops and materials, increased course enrollments, and 
improved leadership activity were- additional outcomes. 
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2. The project staff developed a pre-post assessment instrument / 
and used it to interview key personnel in an experimental- 
control group evaluation design* Thfe data they qollected 
were' first used to assess the instrument's reliability and^ 
validity. The s^me data were analyzed to determine, if 
significant ^act.was made by project-developed strategies 
4 and to develop "prof lies H of successful and less successful 
users of th£se strategies. Results indicated that the 
implementation of project-developed resources in partici- 
pating LEAs enabled these) school districts to make signifi- 
cantly greater changes ttt^a did t hp control districts* 

Evaluations of the othe^ three SEA grant, projects were inconclusive for a 
variety of reasons : 

1. One SEA grant project collected evaiuatioiljflata but lacked 
resources to compile and analyze the data. Impact assessment 
consisted of c^llectipg jige-posfr data on attitude change 
resulting from inseryicesessions that used pro jec developed 
materials. Workshop evaluation questionnaires were &l$g 

.administered. Through an overview? of the evaluation data, , 
project staff had the impression that most of the workshop 
evaluations were "better than average," but that attitude 
change appeared to be minimal* 
■ * * * 

2. In this .SEA grant project, it- was impassible to say what 
part of the changes noted in the evaluation were due to (he 
grant project, because of the variety of social, forces that 
also could have been contributing influences. Evaluation 
data were gathered and analyzed wit^i sophisticated, statis- 
tical techniques. Changes were <£ounci to have occurred in v 
the desired direction, suggesting positive project impact. 

3. Another SEA grantee found that it would not be meaningful to 
use a pre-pos Revaluation design 't<o measure 'the- impact of 

. the WEEAP. materials, as* had been originally proposed. There 
'werfe two ''reasons for this: the groups using any Tone* set of 
WEEAP. materials were 'too smail to yield statistically sig-- % 
~-\ . nif leant results-; and the WEE At materials -formed only a part 

of th& overall equity programs as eventually implemented. 
The evaluation of this grant project teas not completed ,at 
the time of our interview, but it will be based on some gen- 
v - eral pre~post attitude testing,' informal interviews with ■ 
pro ject participants, and staff observations* 

In summary, the evaluations conducted by three of the six SE^As were 
relatively. Sophisticated approaches to determining the impact of using the 
strategies or materials they had developed under their WEEAP grants.' 

SEA I mpre ssions of WEEAP Products Purchased and Their Impact (N-3» SEAs) * 
_ — . ~ > 

The data frpm our SEA interviews complements the user survey .reported 
in Chapter II, although conversations with SEA staff were more general and 
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pertained to any and all WEEAP products purchased rather than tot single . 
product* * • . 

t * * v , What types of products were purchased ? The three SEAs in the- sample ^ 
purchased curriculum materials and staff development materials. No mention 
was made of other types of materials in the WEEAP Publishing Center catalog, 
such as Counseling and guidance, Career Development t Early Childhood Educa- 
tion, or Educational Administration materials. 

How were the materials used? - In all cases, the SEAs used the WEEAP 

VN materials at regional or local school district "show and tell" workshops or 

media, fairs for teachers . The SEAs 1 intent was to demonstrate how to use 
the mdterdals and: to mafce teachers more aware of the educational equity 
resources available .to them\_ Oth^r uses of the WEEAP materials were: 

• to pravide SEA*staff with reference material and/or a "work- % 
ing pro|essional library;" and 

# to provide V^EAs with reproduced copies of parts of the 

* materials..!) i response to their specific requests. 

What were the cechnical arid substantive qualities of the materials ? 
The three SEAs unanimously praised the substantive quality of most\of_ WEEAP 
products they had purchased* In general,' these SEAs were also pleased with 
the products^ technical quality; one mentioned, however, that the technical 
-quality of some videotapes that had been purchased was poor* ^ 

Were the materials easy to implement or to adapt ? In commenting on the 
- general need to use the products flexibly, the three SEAs expressed the - \ 

opinion that WEEAP ^materials are too comprehensive, detailed, and complex to 
be implemented in their .entirety (3/3). They said, for example 5 that most 
LEAs are not able to cocanit themselves to i full course or series of work-- 

shops on educational equity and prefer strategies or materials that either 

» 

ean be infused in an existing curriculum or pulled" apart into shorter les- / * 

5- » 



sons or activities* .Two of the -SEAs commented that most WEEAP materials "are 
easily adapted to local needs; the third felt that while spme of^the materi- 
als assume too much knowledge on the part of LEA users, "they are. well suited 
for use 'by SEA personnel, who are already attuned to equity issues." 

What were the .outcomes from using the materials? , Table 38 shows the . s 
* " outcomes that this .sample of SEAs expected and accomplished from using WEEAP 
materials. Most SEA purchasers felt that their objectives had been met; no 
one said that the materials definitely did not, bring the desired outcomes. 
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Table 38 



' ' OUTCOME'S EXPECTED AND. ACCOMPLISHED BY SEAs ' ' 

•.?*•' \ FROM USING WEEA*P MATERIALS L 

tTN»3; double counting results from multiple expected outcomes) 

' --Jr ' . * * 

Expected Outcomes • Frequency 



Expected 



Accomplished 



Not Sure 



Various state' and local educa- 
tional and /or administrative 
, personnel 'trained in sex-fair 
'curriculum, T^tle IX issues, 
equitable classroom interac- 
tions, -or using equity 
materials with students 

Local school districts (LEAs) 
adopt equitable classroom 
strategies' & infuse curri- 
culum with equity materials 

LEA workshop participants spread^, 
information about equity 
issues to others in LEA 

TOTALS . ' - 



. 3 



-7 
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What indicators were accepted as evidence of these outcomes ? One of the 
three SEAs interviewed had done a formal summative evalliation of .the impact/ 
of an educational equity Workshop program. Each workshop was evaluated with 
a post session "feedback questionnaire." In addition, questionnaires were 
mailed to a sample of workshop participants asking whether knowledge and 
materials acquired were shared with others in the LEA, what the local recep- 
tion was to these' ideas, and what, if any, change in classroom behavior; ' * 
resulted. ^Data from the-4-5 percent of their sample who responded to the 
questionnaire were compiled and analyzed. Results were as follows: 

\ • Almost all (91%) reported some change in classroom behavior 

as a result of the workshop materials. Changes included eli- 
mination^ grouRing by sex, -elimination of sexist language, 
better balance of 'male-female class interaction, and evalua- 
tion of texts and materials for sex fairness. A larg6 pro- 
• . portion (86X)*of respondents felf there had been "a reduction^ 
of sex stereotyping" in the classrooms in their LEAa. v 
* * » 

• A majority of. participants did share knowledge and materials. v 

• A majority reported that LEA. personnel were cooperative and 
receptive to the new ideas arA materials. v 
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' • The other two of these three SEA.piirchasers did not conduct formal 
4 evaluations. In one case,, the SEA staff who visited L&te had the impression 
that only about five percent of the iriservice participants went on to use 
the equity, materials but. t;hat. LE& personnel who used WEEAP materials liked 
thfcm very much. The'second SEA evaluated some workshops, but not othersv 
* This SEA relied mainly on year-end reports submitted by LEAs to estimate the 
impact of the educational equity program. These reports indicated a posi- 
. tive response and the development df local equity projects. 

In summary, it is difficult to draw conclusions about , WEEAP product 
S\ impact per se from this sample , of SEAs, partly because of the small size of 
the sample, partly because the SEAs were deliberately selected for their 
extensive involvement in equity activity, and partly because quantitative , 
data were not commonly cOllecte'd by tlfiis group as ,evidende of:WEEAP product 
impact'or of the impact of the SEAs 1 equity activities. W6 can only say 
that the three SEAs that did conduct formal evaluations were satisfied with 
the JJEEAP" materials and with their own progress in faking their educational 
programs, practices, and policies more equitable. They\planned to continue 
their activities in much the same way as in the past; the one change men- 
tioned was the intention of one SEA to purchase WEEAP materials that -were 
more directly related to subject-specific curricula. * 



Future Directions of Educational Equity Activity .(N-17 interviews in 9 

< ■ ! 7 " 

states) 

!as, a way ol estimating "the past and likely future impact of the WEEAP 

* ♦ * 

program, a sample of 17 WEEAP grantees, product purchasers, and other SEA 
personnel involved in educational equity was queried about the future of 
educational equity in their states.*. Results follow. » 

Are there plans for continued use of WEEAP products and approaches ? 
This issue was approached by discussing two topics: (1) whether the agency 
with which the interviewer was associated* planned to continue its currents 
activities and to use WEEAP materials. .and (2) whether the SEA would continue 
to platan active^role in educational equity. Most of those interviewed 
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* It should be noted that responses represent the* informed opinions of a v 
group df people who are involved in educational equity issues, but-who are * 
not necessarily decisionmakers in their states regarding budgets or pro- 
gram planning. (Refer to Table 33 for a listing of the positions of the 
interviewees.)- , *- ' 
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believed that their SEAs 1 activities in" educational equity will continue in 
the future in some. form (16/17). Specifically, 76% said that'the state 
agency 'in which'they worked will continue to acquire WEEAP products "as long 
as the. -agency's funding continues." They cited varying -degrees of depen- 
dence on WEEAP products,' a's is shown in Table 39. 

' • Table 39' 

DEGREE QF PRESENT. DEPENDENCE ON WEEAP MATERIALS BY STATE 
f AGENCIES PLANNING TO CONTINUE TO USE WEEAP PRODUCTS (N-13) 

Degree of Dependence . Frequency Reportec^ 
Rely heavily on WEEAP products 4 
Use many WEEaP Products 2 

* .Use some WEEAP Products 3 ' 

• Use a small number of WEEAP Products 4 ^ ^1 

c 

No comment % ; ;JL 

TOTAL ' £ v L3 

* 1 / 
A1J. of those interviewed said that their SEA will probably continue some 

level of equity activity but their responses were nearly all qualified. 
For example, 0 the state will continue to fund these activities jtf there is 
vocal 'community support (4/17); i£ federal funding continues (4/17); if , the 
CSSO continued to besupportive (2/17). In the absence of any federal fund- 
ing, as is shown ^in Table 40, few interviewees (4/17) believed that their 
SEAs would continue to support educational equity. 

. . Table 40 

LIKELY LEVEL. OF STATE EDUCATIONAL EQUITY ACTIVITY IN 
* THE ABSENCE OF FEDERAL FUNDING (N-17) 

Level of Activity v > * F requency 

Increased activity* ' 2 

Continue at same level ^ \ 

Reduced level of- activity - 3 



Minimal or gVeatly curtailed activity -7 
Uncertain t % , j 

TOTAL v 17* 



* One of these- expected greater activity with a smaller staff. 



If - federal fundiftg for educational equity is included *in "block grants" 

atp some future lime, what Is the likelihood that the state will allocate the 

present^ level of funds for Educational equity ? Nearly half (8/17) of those 

' * ^jterviewed said that their SEAs would be unlikely to allocate, fund a out of 

block grants' to continue equity activities at the present level (that is\ 

the level of current federal plus state funding). Looking at responseson a 

state-rbyrstate basis (rather than" at individual interviewee responses) we 

f irid that allocation of funds for educational* equity out of block grants Is 

likely to be in j&opardy in six of the nine states. - 

< On the ottifcr 'hand,' five interviewees thought their/ SEAs would continue 

•.to fund all current, equity activities in a "block grant" situation. Two of 

these fc five interviewees believed that block grants would actually improve 

the position of educational equity funding (2/17). These two individuals 

gave different reason^, for a positive prognosis:* 

« In one state, educational equity is riot a top priority and 
there* has been little SEA Interest in seeking out funding, 
for this purpose. However, the individual interviewed hoped . , 
j that bloek grants would have funds "earmarked" for equity, 
and if that were the case, the SEA would have to "use the 
funds for the designated purpose. In addition, "this* inter- 
viewee thought that a well-coordinated state-administered 
program would have more potential for close contact with the 
LEAs statewide than WEEAP's current grant system can provide. 

• The second of the two states had strong legislation, a com- 
s mitted CSSO and legislators., and interested LEAs, but lacked * 

funds for equity activity* Block grants were perceived as 
having the potential for providing the funding this SEA 
heeds to strengthen its existing conaditmeut to Equity. 

If equity materials were' no longer available through WEEAP, would the 
SEA develop-and publish similar materials? There was a consensus on this 
point in o'nly one state, where botH interviewees said the state would defi- 
* nitely not fund the development; of equity materials. Six interviewees 
(representing five of the nine states) sai4^;that such funding was not^ 4 
likely; th^ee (representing three states) said limited funds might be made 
available; five (representing five states) said the states would fund such 
efforts; and three interview rs in'three states declined to give an opinion. 




. What factors encotfrage or discoutage educational equity actlvlty"by 
the SEA? Table 41 displays the following information 'for each of the 
nine states^ x 

indicators of the state's current activity or involvement in 
educational equity; 1 % 

he current role played by the SEA ..with respect to educa-, 
tionaL equity; * " • 

the future likelihood that the'tstate vij.1 commit block: grant 
funds' for^ducational equity activity; and 

the likelihood that the state will develop and publish edu- ' « 

cational* equity materials in the absence of ^such materials 
" from WEEAP. 

Tfie one state (state "C") where there*, was plosest Agreement between 
interviewees that the state would probably fund educational equity out of 
block grants and would probably develop equity materials was also the state 
with the greatest number of indicators of pro-equity* involvement .(5+3-8 . 
indicators) . In the two other states T ;ith considerable equity involvement 
(states "H" and "I"), thqse interviewed were also fairly positive about con- 
tinued funding. At least two of these three ^fcates (C,, H, and I) showed the 

» 

following indications of equity involvement: % 

(1) , The CSSO supports educational equity (3/3). . 

(2) There is a SEA plan for future equity activity "(3/ 3) . 

(3) The state currently provides some funds for this purpose 
(3/3). . m k - ^ 

> 

(4) The SEA* s role has been to provide technical assistance, to 
disseminate information, and to monitor fox: compliance with - 
provisions for .educational equity in LEAs (3/3). 

jt 

(5) There is strong state legislation for equity (2/3). 

The jy.ve states with the least amount oi equity involvement according 
to the eight indicators (states "A," "B," "D, H -and "F^'weie ^Iso thos* 
with the lowest perceived likelihood of future equity funding* (State "G" 
is not Mentioned here because there was only*one interviewee, and thus one 
'response to each question.) ■ 
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Table 41 .» , x ^ 

* t ♦ 

INDICATIONS OF STATE ACTIVITY/ INVOLVEfiENl IN EDUCATIONAL 
EQUITY AND LIKELIHOOD OE FUTURE INVOLVEMENT 
(N-17 interviewees representing 9 states) 



State 



Indicators 



A r B 



Supportive CSSO 
SEA Equity Plan 
Equity Legislation 
Current State Funding 
Political Support 
TOTALS 



SEA Role in* Educational Equity 
Provides Technical Assistance 

v * « — o 

Disseminates Information 

» 

Actively Monitors Compliance 
TOTALS 



XXX 
X 

* X X 

X X 
• X 



F V Q H I TOTAL 

x x x ' " 6 
x x x x 5 
xx 4 
'xx 4 
1 



0 2 5 2 0 1 /3 



3£/ X 
X 



Would state fund ed. equity 1 + P 
out of block grant funds? _3_ 
TOTAL (2 interviewees per state)* 3 



Would stata.develop/publistv + 0 
equity materials? 2 
TOTAL (2 interviewees *per state)* 2 



x 

X 



X 
X 
X 



X 
X 
X 



X 
X 



X 
X 
X 



20 



X 
X 



2 3 . 3 



0 
2 



3 
3 



0 
1 



T7 



(0) 

IzL 



2 1 
3 



Ill (N/A) 5 4 



0 
0 



2 
3 



1 1 (3) 0 1* 
0 0 (-) 3 3 
1 1 (N/A) 3 4 



7 
5 



21 



* Coding: 0-N0 

. A l"Uncertain 

2»0n a Limited Basis 

3-YES 

-■only one interviewee (State "G") 
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Figure 4 was drawn to show the 'correlation between the number of indi- 
cators^ past state involvement in equity (the eight indicators in, Table 41) 
and the .perc'eived likelihood of future state involvement. The fact that 
nearly all states appear within the narrow cigar-shaped band indicates that 
there is definitely a positive relationship: the more indicators of past 
involvement, the gresfter perceived likelihood of future involvement in 
equity activity. 

It 'should be noted that all of the individuals interviewed were person- 
ally committed to the goals of educational equity and have made it the focus 
of their professional careers in recent years. Nevertheless, and despite 
eir general optimism, most (10/17) commented during the interview-* that 
educational equity is not * their states 1 top educational priority and that 
strong competition for shrinking state education funds will be a hindrance 
to future fuading qf educational equity (8/17). For this reason, 
stressed the importance of continuing federal equity programs such as WEEAP. 

What could WEEAP do to assist SEAs more in their educational equity 
efforts? SEAs are f requently \the main source of information, materials, and 
guidance for LEAs on issues of educational equity. For that reason* it is y 
important that WEEAP include the SEAs in its strategies to develop model 
equity products, disseminate them, and demonstrate their value. Interviewees 
in this sample of SEAs wanted WEEAP to take steps to (1) help SEAs be more 
responsive to the needs of LEAs and (2) help SEAs 1 provide better leadership 
in the area of educational equity. Specific suggestions for WEEAP suppor- 

9 

tive action included the following: 

• Sponsor workshops for SEA personnel to familiarize them with 
WEEAP materials. 

• Provide free samples to SEAs of all materials from the WEEAP 



Publishing Center. 




• Fund the development of models of SEA technical assistance 
pertinent to Title IX and other equity issue ^. 

• Develop guidelines for use in reviewing curriculum and 
evaluation instruments for sex-fairness. 

• Develop evaluation instruments to measure the effectiveness 
of aducational equity programs. 

Fund gsant projects to develop materials *in the following 
content areas* judged by SEAs as being in demand by LEAs: 




> Figure 4 

CORRELATION OF PAST AND LIKELY FUTURE INVOLVEMENT IN EDUCATIONAL EQUITY* 
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* Includes data from the' eight states where two people were interviewed. 
Index represents sum of coded responses noted on Table 41 # 



student leadership training; sex-role identity needs of 
teenagers;* sexual harrassment; and domestic violence. 

• Fund grant projects to develop materials intended for use 
with Spanish speakers, community groups, and males. 

' The SEAs also felt the need for more information sharing among all these 
interested in educational equity. They hoped that WEEAP would do some of 
the following to take a lead in improving information sharing: 

• Provide information sheet? on potential funding sources for 
educational equity efforts. 

• Distribute updates to SEAs about educational v equity activity 
at the federal level and among CSSOs. 

* 

• Build a "WEEAP data base" and fund research projects for this 
purpose. Data«of interest to SEAs would include the amount • 
of -money spent to date by WEEAP, numbers of people and agen- 
cies affected by the program, and reliable information on 
the impact of specific WEEAP products and strategies. 

• Distribute bibliographies of sex-fair materials organized by 
subject area. A related suggestion was to fund a clearing- 
house for educational equity publications from all sources, 
not just from WEEAP, 

• Compile a directory of model educational equity programs and 
of resource groups and individuals organized by geographic 
region . 

.# Fund mere actioproriented projects as opposed to product- 
development projects. 

• Sponsor regional conferences where legislators, SEA leaders, 
CSSOs, and others can learn about the value of educational 
equity for improving educational programs-, policies, and 
practices. 

Many of the SEA suggestions for supportive action and information 
sharing by WEEAP are already part of the WEEAP strategy. However, the SEA 
interviewees were apparently not aware of these ongoing activities, a fact 
& which in itself speaks for the necessity of building better WEEAP communi- 
cation with SEAs* 
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f i Summary 

"» * • 

*» 

Results of the survey of educators in nine state education agencies . 

(SEAs) showed that these equity*-involved individuals were pleased with the 

outcomes from their WEEAP grant project activities and from us§ of materials 

developed by other WEEAP grantees. They cited numerous examples of positive 

outcomes that tjhey felt. could be attributed to the WEEAP-fUnded activities 

or products. Rigorous, quantifiable evaluation evidence for these outcomes, 

* 

however, ,was offered in only four of the eight states where information on 
evaluation could be obtained. 

The generai satisfaction with WEEAP-developed materials and strategies 
was apparent from the fact that this sample of SEAs hoped to continue to 
rely on WEEAP as' they had in the past. The importance of a federally funded 
Equity program was also underscored by the finding that most SEAs would 
probably not be able to support out of state funds the kinds of activities 
that WEEAP presently supports. This survey also indicated that those states 
with the greatest current involvement in educational equity activity (whether 
such activity be state- or federally funded) were also those states with the 
greatest perceived likelihood of continued backing for equity activity. 
#; " • 'Finally, the SEAs in this sample had numerous suggestions for ways that 
WEEAP could assiit them in their continuing efforts to improve the fairness 
to both sexes of their states' educational programs, policies, and prac- * 
tices. They also ^suggested specific ways in which WEEAP could take the lead 
in promoting better information sharing among all those interested in educa^ 
tional equity. 
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Implications for^W&EAP's Future Evaluation Planning 

A major WEEAP objective is to provide educational systems with tools 
for improving the equity of their programs, policies, and practices. State 
education agencies (SEAs) are a major channel for reaching local educational 
systems. The SEA perspective, therefore, is an important one to incorporate 
in WEEAP 1 s future plans and in assessments .of the value and impact of its 
products, approaches, and projects. 
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v/ If WEEAP wants to generalize from^the responses of a f v SEAs to all 

SEAs, future evaluation planning must provide for drawing a random sample 

from the universe of SEAs* There is a strong implication in our findings 

that one .basis for selecting a SEA sample would be the presence, or absence 

i » " 

of certain "indicators of involvement in equity •" , 

Questions that our short assessment did not -answer but that could be 

addressed in a future evaluation would include the following: 

* * « 

• What 'kinds of specific requests for particular types of 
equity materials and assistance do SEAs receive from LEAs 
most often? 1 

^ • What specific types of materials do states .develop for them- 
selves? 



What advice can SEAs provide to WEEAP and the WEEAP Publish- 
ing Center on the states' specific preferences for WEEAP f s 
future development, dissemination, and demonstration of its 
products and approaches? 

Are there specific ways in which the SEAs and WEEAP can 
collaborate in improving" the effectiveness and economy of 
promoting and disseminating WEEAP products in LEAs? 
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N Chapter VII 

* ■ INTEGRATED FINDINGS FROM THE RAPID FEEDBACK EVALUATION . 
AND RECOMMENDATIONS FOP. IMPROVING PROGRAM MANAGEMENT AND EVALUATION 



/ Purpose 

In Chapter I, the purposes of the rapid feedback evaluation were set 
forth: 

> # t6 obtain, a rough estimate of progress toward the agreed 

upon objectives for ttfe WEEA Program from information* that 
was easy to obtain and^ analyze^ , 

to estimate the feasibility and likely utility of conducting 
a more formal evaluation of program performance in the 
future; and Si 

• to recommend action steps (management and evaluation options) 
which could be exercised to improve the program. . 

In this chapter, we deal only with the first and third of the above 
items. The second item was addressed at the end of each of the preceding 
chapters in concluding sections called "Implications for WEEAP f s Future 
Evaluation Planning/ to which the reader may refer. 

The first section of this final chapter integrates results summarized 
-in the preceding five chapters. The summary of integrated findings indi- 
* cates the progress WE EAP has made toward the agreed upon objectives for it 
field operations. ' 

The second section presents management atid evaluation options which c 
be implemented to strengthen the WEEA Program and to improve its ability t 
demonstrate measurable success in achieving its objectives. 



% Integrated Findings 

WEEAE's Objective: "Develop diverse, pested model nroducts and "chpitge 

strategies." / 

Findings on "Diversity" of .WEEAP Products 

• « < 

• A variety of materials have in fact been jdeveloped under WEEAP 
grants; most of these were in* three major categories: Staff 

> Development; Curriculum; and Career. Development* 

* Our analyses of sales data suggested that, items in the Staff 
Development and Curriculum categories may be approaching the 
over-supply point — at least for the particular items being 
offered: Greater selectivity in future funding of product 
development in these categories r (and in pre-publication 
screening reviews) may be justified. 

# Conversely, there may be justification for increasing the 
relative proportion of Items in the Career Development and 
Guidance categories, 

• Even though there were 193 items available for sale through 
HEEAF's Publishing Center and these, consisted of a diverse 
artay of products .anA approaches , our interviews with users 
indicated that gaps and unmet needs existed** Examples included: 

- the need for more .materials designed specifically for 
classroom use and staff development at the elementary 
level, the postsecondary level, junior high school, and 
high school; 

• more products which are amenable for use with males as 
well as females; 

- more products which inciude short, very easily adapted and 
easily infused activities; . 

j 

- the need for materials which build in short lesson-plans 
and/or which link parts of the material to topics, themes, 
and subjects commonly taught in' school; 

- more products for use with or by disabled,'* minority,* and 
re-entry women; 



* The number of products and approches which address the needs of disabled 
axjd minority women can be expected to increase, because WEEAP established 
priorities for such projects in its 1980 regulations and has increased the 
proportion of its budget to be allocated to these* two priority areas in 
e^ch year, from 253! in FY f 80 to 40% in FY f 81 for H racial' and ethnic 
minority women and girls'* and from 10% in FY f 80 to 15Z in FY '81 for " 
"disabled women and^girls." 
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— the need for model strategies for use by agencies such 
as state education agencies (SEAs) in providing equity- 
related technical assistance; 

- the need for more materials which are designed to be used 
with community groups* 

: n ' ■ " 

Findings on "Testing" by Grantees of Products They Develop for WEEAP s 

* 

+ Most grantees •'tested*' In so\ne way the products they devel* m% 
w ' oped* However, their evaluations were limited in scope and 
lacked rigor because of the small budgets and short time 
lines N of many of the grant projects and the limited evalua- \ 
tion expertise of their staffa. , 

• Appropriately enough,- grantees' evaluations* focused on data 
of immediate Interest to N them in successfully producing 
material that would be judged worthy. of dissemination by 
WEEAP. While grantees' evaluation objectives occasionally 
overlapped, with the broader programmatic objectives qf most 
interest to WEEAP, this was unintentional* 

• Grantees were virtually unaware of WEEAP f s overall objec- 
tives, the outcomes of grant projects that would be of most 
.interest, and how WEEAP would actually make use of such 
information if grantees could provide* it. 

t 

• Grantees reported that their inability to demonstrate more 
dramatic change was the result of methodological or design 
problems in their evaluations and not to weaknesses in their 
products* . < 

• Grantees were unsure about where to go for help with their 
evaluation problems, which they judged to be too tough to • 
resolve easil^* within their grants' budgetary and time con- 
straints* 

* *^ ' ■ 
e As a group , the SEAs in our sample of WEEAP grantees were : 
more likely to have done sophisticated product and project 
evaluations* (SEAs probably had easier access to the exper- 
tise required to do good product and project evaluations.) 

0 

Other Findings on jPgoduct Development. 

• The substanti/e and technical quality of WEEAP products 
overall were perceived as having improved since products 
first became available through Jie WEEAP Publishing Center 
late in 1978. ' 
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• Many interviewees perceived that there was room for further, 
improvement in audiovisual materials* ^ . \ 



A 



WEEAP's Objective': /'Prodi^ce and market the best of these model products 

and strategies to potential users, nationwide/ 1 

i • 
Findings on Production of WEEAP Materials by the Publishing Center 

e The cumulative total of grantee-d.eveloped materials available 
from thw WEEAP Publishing tenter more than doubled between 
1979 and 1980 (from 69 items to 140j based on items listed 
* im the Publishing Centers cumulative sales Records for 

those two years) . \ ,/ > 

. - ■ ■ * 

• The. major types of materials that passed the screening . 
review and were .published as of th^ end of\1980 were Staff « 
Development, Curriculum, and Career Development materials* 
These .three item types accounted for 90% of the products 
available for sale by the WEEAP Publishing Center. 

• In 1981, more than two-thirds (6 9%) of the 193 items avail- 
able from the Publishing Center cost less than $10.00 and 
43% cost less" than'i5.00. 

• .While moderately .priced it6ms sold better than expensive 
items (those over JlfoOO), this may not have indicated a 
lack .of popularity "but ra.ther the option to obtain many of 

' the expensive items at a much lower price through rental. 
For example, all 34 audiovisual items in the "expensive cate- 
gory could be rented* , * , 

• Several of the grantees in our study -whose materials had been 
produced and marketed by the WEEAP Publishing Center wante.d 

.the lag time reduced between submission of materials for > 
pre-publication screening review and production/marketitig ^of 
materials that were approved for dissemination. 



Findings on Procedures for Marketing WEEAP Products 

• Some of the grantees in our study whose materials had been 
disseminated by the WEEAP Publishing Center were concerned 
that marketing was not tailored sufficiently to potential . 
users for specific kinds of materials (within major cate- 
gories used to sort them for the WEEAP Publishing 'Center 1 8 
catalog) . . x 
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^ # • Grantees had received requests for their products from a 

* wide variety off individuals and groups and could make these ^ 
\ known to the WEEAP Publishing Center for /use in future mar- 

keting of the grantee's own and dimilar products. % * 

' • Data from our grantee -and user samples suggested that WEEAP 

4 consider the potential valute of face^to-;face contact between 

product developers >and users as a way to enhance sales and • f 

to increase the likelihood of product utilization. 

« *• Several product users in our samples expressed the opinion^ 

that dissemination'could pe improved by offering more oppor- 4 „ 

tunities for potential purchasers to examine materials v 
first-hand. v 

• v 

# Many of the product? users in this study were N unaware, that. . . 
*. ■ % development of the items the^ purchased had been funded by 

WEEAP. Thip connection 'escaped them in spite of. the stan- 
dard acknowledgment included in the materials and the Pub- 
lishing Center's efforts to highlight WEEAP's role in sup- * 
porting product development and dissemination. 

^ Findings on Outcomes from* Marketing* WEEAP Products 

* ,» 
* * 0 f A. total of 42,798 items, representing $333,866'in dMlar 

volume, were sold by the WEEAP Publishing Center from 1978 
through 19130. (The first sales were recorded during the 
v . latter part of .1978.0 - . - W 

# WEEAP product sales more than doubled between 1979 and 1980 
tin terms of the number of items sold: 12,112 items were 

sold in 1979 and 28,940 items Were sold in 1980: The dollar 
i ' volijme represented by these figures was $119*,117' in 1979 and 

$184,256 in 1980. 

. * • " The WEEAP Publishing Center appeared to.be marketing, prod- • 

uctV successfully to potential users for whom, th^se products 
v were originally intended. Our estimate was that between 82% 
„ and 92* of WEEAP products were purchased bjj postsecondary /' • * 
institutions,, state education agencies (SEAs), and jelemen- , . 
tary /secondary institutions. - * 

0 Although" all types of purchasers bought a wide variety of 
items, most bought Staff Development, Curriculum^ and Career 
* Development .materials. Most of the available products ar,e 

in these three categories and accounted for about 902 of, the 
' Publishing Center's overall sales -of WEEAP products from 
1973 through 1980. , % 

• /WEEAP ^products appeared also to have modest appeal for types - 
\ *of purchasers for which they were not originally envisioned 

/ or planned*. Our data .indicate^ that purchaser types who had / 
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a not 'been the Publishing Center's primary audiferce for parti- 

cular types of products were buying them nevertheless (e.g., 
: businesses, professional. organizations, women's centers, and 

individuals) • , 

i • * 

Every state in the nation is represented, in the WE SAP Pub- 
lishing Center's file of purchase invoices, but our prelimi- 
nary data suggested that, there were substantial regional x 
variations in the quantity and dollar volume of items pur- 
chased and in the types of buyers who were purchasing these • 
items. There were no regional variations in the major types , 
of items purchased, however; all' regions bought mostly those 
itemd in the three top-selling categories.. J 
• * * - i 

A Finding Related to Expanding WEEAP* s Dissemination Focus in the 
Future \ 

Several individuals, including those in our sample of SEAs, suggested 
that WEEAP now expand its dissemination focus beyond providing "tools" like 

model products and approaches* They cited specific needs for: ^ 

, «■ 

• disseminating various kinds of information, such as current 
equity^ctivities in other agencies, other source? of equity 
materials,*, and evaluation data indicating the impact . of 
equity projects and materials; 

• facilitating formation of "working networks" by providing 
a wpy to locate individuals in similar equity-related 
endeavors; 

• 1 
4 making known various sources, of funding for initiating or 

continuing equity activities. 
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WEEAP 1 s Objective: "Demonstrate. that these model products and strategies 

are usable, useful, valuable, beneficial, and adaptable 



"Specifically, demonstrate that use: 

"produces positive changes in participants 1 beha- 
vior, attitudes, aspirations, and awareness and in 
educational policy and practice; • 

"fosters a conducive environment for equitable 
change ; 
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' "enhances the capability of the educational system 

to achieve educational equity." 

Findings on Users 1 Choice of WEEAP' Products 

• WEEAP products, were consistently chosen for perceived ease 
of use. For- example, curriculum materials with short,, 
easily adapted and easily infused activities were the most 
sought. after by staff at the national demonstration sites. 

♦A related finding was that purchasers looked for products 
that appeared ttf be easy to use in part, rather than in 
their entirety, and that could be modified without a major 
curriculum writing effort. -» 

• The products; were most frequently chosen with the intention 
of. using them for staff development, for students T classroom 
activities, for background for developing curricula or work- 
shops, or as an onrthe-shelf reference for staff. 



„ Findings on Utilization of WEEAP- Products 

•< WEEAP products were judged to be self-explanatory, easy to 

use, and easy to adapt. * 

v 

• WEEAP materials were rarely iJsed intact. Instead, parts of ✓ 
a product were selected and adapted to su't the user T s' 
situation. 

Use of WEEAP materials did not require special technical 
* assistances This finding was not surprising, given that 

products were chosen for their perceived *ase of u,se and 
that users freely selected and adapted pbrtions of a product 



they wanted, to implement..^ 



t 



Staff at the five national demonstration sites unanimously 
reported that ins#rvice training which focused on particular 
WEEAP products led directly to utilization of those same 
products by the workshop participants. . ^ 

WEEAP products were frequently used in combination with 
other resources. While thjji syggested that these. WEEAP 
products were flexible enough to be integrated easily with 
other materials and presentations, it ruJad out the possi- j 
bility of attributing the event's success dire'etly to WEEAP 
products and approaches. 
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Findings on Outcomes from WEEAP Product Utilization/Demonstration 



Grantees and product users accomplished, to their satisfac- 
tion, most of the butcomes they expected The most fre- 
quently reported expected outcomes were as follows: to 
develop useful products and approaches; to Increase aware- 
ness of or Jto achieve positive attitude change toward edu- 
cational equity issues and women* s issues on the part of 
students, educators, and administrators; to train educa- , 
tional and administrative personnel in equitjr-related 
skills, approaches, and product utilization; and to initiate 
or ^Implement equity activity, materials, and projects* 

Very , few users or grantees had quantifiable evaluation data' 
that could .specifically and reliably support^ the above 
claims. $ 



The WEEAP grantees in our samples identified two major areas 
of contribution to the overall quality of educational pro 
grams, policies, and practices: J 



- increased awareness of women's educational equity 
issues; and s 

4 - incorporation of the 4 grantees' product(s) into 
existing or new courses and programs* 

f . t 

■ • Demonstration site staff were positive about the results of 
this strategy, as the first full year of integrating WEEAP 
products and approaches with existing educational programs in 
their school districts was ending and funding for a second 
year vas assured* A report that--to the extent possible — 
provided a coordinate'd view o/ overlapping evaluation find- 
-^Lr%s at the five sites will be available next year.. 

• The SEA grantees in our sample reported continued, related, 
post-grant activity. affecting educational policies and prac- 
tices within the SEA and throughout the state. 

Findings on Continued Equity Activity in the Absence of WEEAP Funding 

• Data from interviews with the SEAs in our samples indicated 
that federal support in the form of infonjiation, n^terials, 
and funds would be required to sustain their present level 
of equity Activity in the- future. * 

* *r 

• We found a strong positive relationship between the number 
of indicators of past SEA involv&ieAt in equity and the 
perceive d s likelihood of future involvement in the absence of 
federal funding ^for this specific purpose* These iritt lea tors 
included: 
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- stropg CSSO support for educational equity; 
f --an SEA plan tor future equity activity; 

- current state -funding for equity activity; 

■> - an SEA role in providing equity-related technical assis- 
tance, information dissemination, and monitoring; and 

- strong state legislation and political support for 
equity • 

Staff at the five national demonstration sites predicted that, 
if WEEAF funding for the national demonstrations ends, educa- 
tional equity activities and utilization of the best of the 
WEEAP products will continue trough the teachers who have 
been trained* However, staffNtere not optimistic about sus- 
taining the current level of effort when'WEEAP funding ends. 



Recommendations for Improving Program Management and Evaluation 

The remainder of this chapter presents action steps which could „be 
taken to- improve the WEEA Program. Recommendations in the first set of 
action steos are categorized as "evaluation options.** The emphasis in each"' 
of these' options is to find out how the program is operating so that strate- 
gies can be shifted or changed. In this .connection, data collection, syn- 
thesis, and analysis may be required. Recommendations in the second set are 
termed "management options." These options call for changes in program 
strategies or operations that are not intended to ba preceded by systematic 
assessments requiring data collection and analysis and outside expertise. 

* The distinction between -management afid evaluation options is made in 
consideration of the facf that program manifcgement will«weigh the advantages 

of exercising certain^ options against the costs of doing so. Management r 

» * 

options, for the .most part, can be exercised without increasing the size of 
federal program stiff, and without the need for outside consultants. Evalua- 
tion options, bnJt&e: other hand, may require supplementing expertise of fed- 
eral program staff with .outside evaluation expertise. . ^ 

Each of thfc two sets of options is further subdivided according to the 
three major components of WEEAP f s strategy for achieving its > bjectives:' 
(1) product development; (2) product dissemination; and (3) product utiliza- 



tion and the demonstration of impact. 
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Evaluation Options 

Product development* ^ . ^ 

• Systematically examine the specifier nature of gaps within- 
major typesT-of items/ as well as possible needs for other ' 
major itei* types i 'The pjroduct of this effort would be speci- 
fications for the kinds of materials which need to be devpl-* • 
oped to fill .the' gaps* 

• Determine, the amount and type of technical assistance the 
Publi8hiftg Center provides to grantees during product devel- 
opment, end Asseae the extent to which such assistance is 
improving the quality of submitted products* 

Product dissemination * 

• Determine whether and how t^he screening review process can be 
.. v * expedited without sacrificing its value or ignoring criteria 
^ ' that .have bee)ji^stablished .for the review panels to use in 

identifying* and^apiiroving the "best" of the submitted prod- 
ucts for, publication and dissemination* For example, prod- 
ucts that are subjected to extensive review, trial, and 
revision" during the grant period and that are clearly out- 
standing in the judgment of WEEAP sta* f and the Publishing 
Centex:* might bypafrg^eview panels in the interests of short- 
ening production timt* Also, products with especially criti- 
cal H tlme value H cou^d be given special review treatment to 
assure that they are not out of date soon after publication. 

• Determine -whether the Publishing Center's technical assistance 
during product development^has contributed to shortening pro- 
duction schedules > in exactly what respects improvements are 
evident K and the nature of effective assistance* The product 
of this effort wouLl.be specific recommendations for refining 
future technical wSffstance during product development to 
achieve further reductions* in lag time and publication cost* 

* . • Assess the appropriateness and cost-benefits to WEEAP of 

utilizing alternative dissemination sources. Specifically, 
include in the assessment of each pptential alternative 
dissemination source the characteristics of products it is 
well-equipped to publish and market, the target groups it 
serves, and the financial implications of utilizing it* 

• The WEEAP Publishing Center i£ succeeding in selling the 
greatest volume to the'major types of purchasers for whom the 
major types of W&JAP products are intended. However, deter- 
mine whether marketing can reasonably be improved to sharpen 
targeting of particular products within t^hese major types. 

Continue to orient marketing toward groups for which WEEAP' 
products are intended, but assess the cost benefits of stra- 
tegies for reducing apparent regional imbalances. Specif i- 
q cally, target for SEAs in states with many indicators of past 
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involvement in equity but that have not been heavy purchasers 
of WEEAP products. Good SEA candidate* for targeted market- 
ing can be identified by reviewing state files at the Resource 
Center for Sex Equity at the Council of Chief State School 
Officers 1 headquarters in Washington, D.C. 

Determine whether products intended for users outside edu- 
cational institutions are selling as well as expected. 
Examples would be product^ intended for use by businesses, 
professional organizations,- and women 1 * organizations. 

Re-assess the c/st-benefits of providing for a computerized _ 
billing systej/so that the WEEAP Publish? Center can con- 
tinuous ly^moni tor sales, performance without diverting staff 
time from the demands of product review, revision, publica- 
tion, and marketing. <: 

If option for providing for computerizing the Publishing 
Center's sales records is exercise^, explore low-cost options 
for obtaining qualitative information needed to better inter- ti 
pret quantitative analyses. One idea- is to pre-print a few 
pertinent questions on the catalog order form. 

\ 

• If the sales records are computerized, also re-evaluate the 
classification" scheme for sorting WEEAP products. For the 
purpose of sales analyses, new categories are needed that are • 
directly pertinent to 'the questions of most interest in plan- 
ning for future product development, dissemination, and 
utilization. The categories that appear in the WEEAP Pub- 
lishing Centerjs catalog at present are not adequate for this 
purpose, although they may. serve marketing purposes. 

Product utlizatiott and demonstration of impact . 

• Once WEEAP has reviewed evaluation reports related to the 
five national demonstrations, explore ways of capitalizing on 
evidence of positive changes in educational programs, poli- 
cies, and practices at the participating sites. 

• Determine which state? have been purchasing WEEAP products in 
amounts well below what might be expected based on current . 
population statistics. Also determine which of these states 
have a large number of indicators of past equity involvement 
and whose SEAs have excellent in-house expertise in evalua- 
tion, adequate data' processing capability, and good publica- 
tion facilities. Then, consider giving preference to SEAs in 
these states in awarding grants for product development and 
product impact evaluations.* The^ rationale is to bring about 
a better balance in regional sales by promoting WEEAP in 
those segments of the market where the potential for sales 
'apparently is not being achieved. At the same time, ^selec- 
tivity based on the suggested criteria could "extend" the 
resources available to the grant project.- 
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• Fund follow-up evaluations to identify continuing effects of 
major WEEAP product utilization efforts, such as the five 
national demonstrations of educational equity in which WEEAP 
products are a centerpiece* 

• Consider selectively funding grantees or contractors to do 
well-designed studies in which the experimental treatment or 
intervention relies on a specific WEEAP product or combina- 
tion of WEEAP products and approaches* Fund' these evalua- 
tion grants/contracts adequately and specify exactly what 
outcomes,, indicators, and information are of^most interest 
to program management and how the information will be used. 
Further, WEEAP should confirm that the measures and indi- 
cators to be used are directly related to its programmatic 
objectives and will contribute substantially to building a 
research base. 

Management Options 

Product development * 
» 

• With a large number of items currently available for sale by 
the Publishing Center, WEEAP should now be more selective in 
funding product development grants. Aim for high-quality 

' items which are most needed . 

m Involve representatives from postsecondary institutions, 
SEAs, demonstration sites, and elementary /secondary institu- 
tions in determining the characteristics of materials that 
are needed to fill gaps in the existing WEEAP product line. 
(These groups should be involved because they are presently 
the major types of users of WEEAP products.) 

• During product development and revision — that is, throughout 
the grant period — implement procedures for determining that 
grantees are bearing in mind sufficiently the eventual use 
and users for their products « Specifically, be sure grantees 
are building in appropriate features that the results of 
this study indicated were denired by users: lesson plans; 
short activities; lots of visuals; how-to information that 
facilitates infusing material into various subject areas, 
topics, themes, and events; suggestions for adapting materi- 
al without a major rewriting effort; use of formats that 
make it easy to use material selectively (to pull 

pieces,, for 4 example) . 



* Encourage grantees to include in their products outlines for 
training others to use the materials. Potential users could 
be further assisted in product utilization and adaptation if 
grantees would include in their materials information for 
contacting resource persons and information on how the prod- 
uct was used during its developmental tryout. 
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• Assess the potential payoff vfrom collaborating with product 
development efforts of other agencies or groups whose objec- 
tives are compatible with WEEAP's. Advantages-: conserva- 
tion of resources while extending WEEAP's arena of influ- 
ence. Examples of agencies or groups are the Council of 
Chief State School Of fibers, state education agencies, vari- 
ous sex equity organizations, networks, and coalitions* 

i 

• Reconsider the existing heavy emphasis on product development 
as the intended outcome of every grant* Selected grantees 
could be funded for other activities: major evaluation 
efforts required to »build a research base; comprehensive, 
actionroriented efforts; demonstrations of collaborative 
arrangements with potential for low-cost training and imple- 
mentation of equity concepts and strategies; networking - 
etc. (WEEAP would ^have to publicize these efforts with new 
dissemination strategies, however*) 

Product dissemination . 

• Encourage grantees to consider alternate dissemination 
sources under certain circumstances, e*g*, their products 
would take much longer to publish for some reason if the 
Publishing Center were to do it; excessive publication costs 
relative to costs of producing other materials would be 
incurred (as for audiovisual items); similar products of 
comparable quality are already being disseminated by the 
Publishing Center, t w c. 

• Consider increasing regional conference and workshop parti- 
cipation to capitalize on such occasions to provide oppor- 
tunities for potential users to interface 4 with product 
developers or product users* The purpose of such face-to- 
face contact would be to enhance product sales and utiliza- 
tion by providing how-to consultation on adapting particular 

WEEAP products to participants' situations. 

* v 

• The Publishing Center currently prints an acknowledgment of, 
WEEAP funding in evelry product and promotes the visibility 
of the federal program in all its promotional campaigns and 
appearances* In spite of their efforts, many purchasers of 
WEEAP products are unaware of the link. This suggests that 
WEEAP try other strategies if it desires to highlight its 
sponsorship of the ^development and dissemination of products 
available from the Publishing Center. 

• WEHAP product promotion should highlight features of prod- 
ucts that this study suggests will appeal to potential \ 
users: very easily adapted and infused material; sample 
lesson plans; guidelines for inservice training in product 

nxse and adaptation; inclusion of appealing ar*d appropriate 
visuals; ease of linking to topics, subject areas, themes, 
and event 3, etc. 
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• Explore the feasibility of using "satellite" groups or indi- 
viduals for presenting WEEAP products at conferences. Sev- 
eral former and present grantees, through their other, equity- 
related activities, have become expert in working with state 
and local educational^ agencies and might pxovide a dissemina- 
tion service to WEEAP free or at nominal cost. 

• Continue to strive to keep prices of WEEAP products low, 
since items under $10.00 clearly sell well, and those under 
$5.00 sell even better. 

• Consider increasing the use of free samples, long-term loans, 
and ptte-purchase examination copies as enhancements for WEEAP 
product sales* Postsecondary institutions ancl SEAs in states 
with many indicators of past involvement in equity would be* 
among the prime candidates for such inducements. WEEAP could 
attach a string, such as collaborative staff development work- 
shops sponsored by the SEA or the postsecondary institution. 
The purpose of such workshops would be to familiarize staff 
with WEEAP products and approaches and to train them to use 
and adapt particular materials to suit their special needs. 

• Reconsider the use of product-line brochures, .eliminated when 
the Publishing Center's contract was negotiated. The bro- 
chures 1 purpose would be to target particular kinds of prod- 
tacts for particular kinds of potential users within the major 
product and purchaser categories. 

• Reconsider the advisability and appropriateness of dissemi- 
nating bore than model products, approaches, and materials. 
Specifically, coiwider disseminating types of information ■ 
thatJtre pertinent tp meeting equity nseds identified in this 
study: information on other sources of equity resources; 
links to equity networks and to individuals actively working 
La various equity areas; leads to other funding sources for 
equity activity; leads to sources of technical assistance — 
particularly in training educators and administrators to pro- 
mote and implement equitable. change within -their organiza- ■ 
tionB and programs* / 

• Selectively fund SEAs, or collaborate with them, in staff 
development activities that utilize WEEAP presets and 
approaches in training local school district instructional 
and administrative personnel Such SEA grants or collabora- 
tion should provide for assessing outcomes of interest to 
WEEAP in addition to. outcomes of interest to the SEAs and the 
participating local district^. - 

• Help SEAs coordinate their current and future educational 
equity activities* WEEAP 1 s purposes would be to: 

-•help SEAs avoid unnecessary duplication of effort in 
product development and inservice training; 
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- extend to all states the availability of materials and 
approaches developed and disseminated in one state; 

- assist states in establishing regional collaborative 
arrangements that will "extend" the resources any one state 
can devote to supporting educational equity activity. 

Product utilization and demonstration of impact . 

• Improve the feasibility. and utility of grantees 1 evaluations 
of their products and projects by doing the following: 

- Limit the requirement for product testing to small "devel- 
opmental try outs" using a sample representing the type(s) - 
of users fpr whom the draft products are intended. 

- Encourage grantees to include low-cost, easy-to-document 
outcomes of interest to 'WEEAP in their evaluation reports. 

- Tell grantees what outcomes are of interest to WEEAP and 

provide them with* examples of, suitable, readily obtainable 

indicators. H 

i 

- Specifically, encourage grantees to document "ripple 

effects" that are directly attributable to their WEEAP- 

funded grant activities and that can be compelling when 

* corroborated by similar findings from other grantees 1 
evaluations of their products and projects. Examples of 
the ripple effects reported in this study included incor- 
poration of grantee-developed materials and approaches into 
existing prpgrams and courses, success in winning support 
for post-grant activity, and requests f ©retraining others 
in the use of WEEAP materials and approaches developed 
during the grant projects. * 

* Publicize activities and outcomes that are directly attribut- 
able to utilizing WEEAP products and approaches at the five 
national demonstration sites. Consider involving the present 

^Director, of the Resource Center for Sex Equity in preparing 
the proposed publication. The Director's knowledge of the 
status of equity involvement in each state could be utilized 
to tailor publicity about the demonstration sites to the cur- 
rent eqyity concerns of SEAs, local school districts, and the 

'state's postsecondary institutions. * ) 
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The evaluation and management options presented in this chapter were 
discussed with program management in the course of reviewing preliminary 
results from the rapid feedback evaluation and in critiquing and revising 
the draft of ^his report. Final decisions on which and Wbw many of the 
recommended steps to take, and in what order, will depend on the WEEAP 
staff's estimate of future funding levels for th^ program, the prognosis for 
a federal role in sponsoring development and dissemination of educational ^ 
approaches, and the potential for strengthening the persuasiveness of pro- T 
gram constituents by providing them with timely and Reliable information on 
the results of WEEAP f s support for model product development, dissemination, 
and utilization. p 
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APPENDIX 



Products Purchased That Were Discussed by - 
Interviewees in the User Survey (Chapter II) 

Positions of Staff Interviewed at the Five 
National Demonstration Sites (Chapter V) 
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PRODUCTS PURCHASED THAT WERE DISCUSSED 
•BY INTERVIEWEES 'iN THE 'USER SURVEY 

• ( ■■■■ ■ V 



The Whole Person Book (4) 

Sex Discrimination in Schools (4) • 

New Pioneers (2) 

How Women Find Jobs (2) 

Developing Women* s Programs (2) 

Sfe* Stereotyping in fftucation (2) 

New Directions, for Rural Women 

Venture Beyond Stereo types 

Trabajamos ' 

Critical Events Shaping Woman's Identity 

Career Shopper's Guide" ^ 

Becoming Sex. Fair . * 

ACTIVE n 

Project CHOICE 

Freedom for Individual Development 
ASPIRE 1 ■ 
Breaking the Silence 
Competence is for Everyone % 
Beyond Pictures and Pronouns 
.Sources of Strength 
New Directions for Rural Women 
Continuing Education for Women 
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POSITIONS OF STAFF INTERVIEWED AT THE FIVE NATIONAL DEMONSTRATION SITES 



1. 



2. 



3. 



Broward County , Florida v 
Project Director - K - 

Project Coordinator 

Lincoln County, Oregon 
Project Director 
Field Coordinator 
Evaluator 

Assistant Sex Equity Specialist 

Quincy, Massachusetts 
Project Director 
Site Coordinator 



4 . Reidsville,~ North Carolina 



Ch 
Pro 



irfnge Faci 
oject Man 



Facilitator 
Manager 



5* Tucson, Arizona 
Project Director 
Secondary Sex Equity Specialist 

s. 
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